“Evaluate the Advancing Systems Improvements to
Support Targets for Healthy People 2010 (ASIST2010) Program”
OMB Clearance Application
Office on Women’s Health
July 23, 2009
Submitted by:
NORC at the University of Chicago
4350 East-West Highway, Suite 800
Bethesda, MD 20814
(301) 634- 9300
Office on Women’s Health
Department of Health and Human Services
200 Independence Avenue, S.W. 537-H
Washington, DC 20201
Supporting Statement for Evaluation of the Advancing System Improvements to Support Targets for Healthy People 2010
A. Justification
Circumstances Making the Collection of Information Necessary
Advancing System Improvements to Support Targets for Healthy People 2010 (ASIST2010) is a three-year, cooperative agreement program using a public health systems approach to improve performance on two or more of seven Healthy People 2010 focus areas. Established by the Office on Women’s Health (OWH), ASIST2010 enables thirteen diverse organizations to use a public health systems approach—defined as an established collaborative partnership between governmental and non-governmental partners—with a sex and gender focus to improve performance on Healthy People 2010 focus areas. Specifically, ASIST2010 programs were charged with improving performance on two or more of the seven Healthy People 2010 (HP 2010) focus areas targeted by this initiative – cancer, diabetes, heart disease and stroke, access to quality health services, educational and community-based programs, nutrition and overweight, and physical activity and fitness – and on at least one objective within each focus area.
The ASIST2010 program builds on OWH’s 17-year history of working to reduce health disparities between women and men, between girls and boys, and among populations of women. OWH has supported model programs and innovations to provide comprehensive, interdisciplinary, integrated health services to women throughout the lifespan. This work has involved model initiatives in academic centers and communities throughout the United States. Early experiences with the National Centers of Excellence in Women’s Health (CoEs) led OWH to conclude that community involvement and the development of community partnerships were critical to advance women’s health. Additionally, in 2007, OWH commissioned a review of the literature on effective sex- and gender-based models of care.1 This report concluded that OWH should build on the success of the CoE model to move toward a gender-based model of health care. Responding to these recommendations, and continuing its previous work in using a public health systems approach to improve health outcomes, OWH launched ASIST2010 in 2007. ASIST2010 differs from previous OWH Multidisciplinary Health Model sites in two key respects. First, it places emphasis on sex- and gender-based care and public health systems approach. Second, ASIST2010 is specifically geared to improve health outcomes within the selected HP 2010 focus areas.
The goal of this assessment is to determine whether use of a public health systems/collaborative partnership approach that adds a sex and gender focus has helped grantees to meet their Healthy People 2010 targets. This study will identify examples of effective strategies and approaches to using public health systems/ collaborative partnerships that add a sex and gender focus to the selected HP 2010 focus areas. The study will explore grantees’ use of evidence-based strategies to implement their programs. OWH is also particularly interested in the extent to which the following outcomes were achieved: the development of sex- and gender-focused strategies; the enhancement of grantees’ existing surveillance and information systems to track progress; the expansion of the public health system/collaborative partnerships; movement towards meeting targeted HP 2010 objectives; and the development and implementation of a sustainability plan. Finally, the study will identify barriers encountered, best practices, and lessons learned. The main research questions include:
Have the grantees incorporated structures necessary to positively impact their selected HP 2010 targets?
Have the grantees implemented processes that will improve outcomes for their selected targets?
What were the grantees’ experiences in carrying out the core activities of ASIST2010?
Is there evidence of changes in outcomes as a result of these enhanced structures and processes?
NORC at the University of Chicago is conducting this study. OWH is seeking approval from the Office of Management and Budget (OMB) to collect data from the thirteen ASIST2010 grantees. Specifically, NORC will conduct the following activities:
Semi-structured telephone interviews with ASIST2010 grant directors and relevant staff members. Two rounds of telephone interviews will be conducted with the directors and up to four staff members from each of the 13 grantee organizations—the first round will be conducted during the middle of the grant, and the second towards the end of the grant. Topics will include experiences with ASIST2010 program activities and progress to date; grantees’ program activities; the impact of using a public health system/ collaborative partnership approach with a sex and gender focus; best practices; consumer satisfaction with the services delivered; and future plans.
Site visits to the thirteen ASIST2010 grantees. Site visits will include the following data collection activities:
In-person interviews with ASIST2010 staff members. NORC will conduct in-person interviews with grantee staff members (including directors and up to four staff members) at each site, for a total of 65 interviews.
In-person interviews with representatives from ASIST2010 partner organizations. In-person interviews will be conducted with up to 52 representatives from ASIST2010 partner organizations (up to four partners for each of the thirteen grantees) while NORC staff members are on-site. The exact number of interviews will vary depending on the ASIST2010 grantee.
Focus Groups with Consumers. Focus groups will be conducted at four of the ASIST2010 program sites with up to ten participants at each site to learn about consumer satisfaction with program services.
In-person interviews with selected consumer representatives. During site visits, interviews will be conducted with up to four consumers at each of the nine sites not selected for focus groups.
Semi-Structured telephone interviews with comparison organizations. NORC will conduct semi-structured telephone interviews with up to ten organizations that are not involved with ASIST2010 to learn about the benefit of designing programs using a public health system/collaborative partnership approach with and without a sex and gender focus.
In addition to primary data collection activities, NORC is also conducting a document review of grantee applications and progress reports to provide context for the assessment. NORC is also conducting secondary data analysis of grantee data from the Behavioral Risk Factor Surveillance Survey (BRFSS) to provide context for the population served in each ASIST2010 location, and their program activities.
This collection of data is authorized by Section 301 of the Public Health Service Act (42 U.S.C.241). A copy of this legislation can be found in Attachment A.
Purpose and Use of Information Collection
This study supports OWH’s goal to foster improvements in health outcomes using a public health systems approach. OWH believes that a public health system/collaborative partnership approach, with a sex and gender focus, will help improve the health of U.S. women and girls as well as men and boys, improve the quality of care, reduce disparities, and potentially, reduce health care costs. This study will examine the ASIST2010 grantees’ programs and assess whether usage of a public health system/ collaborative partnership approach with a sex and gender focus helps grantees to improve performance on Healthy People 2010 (HP 2010) objectives that target women and/or men. This study will develop new evidence about the value of a public health system/collaborative partnership approach with a sex and gender focus on health outcomes. The results of the assessment will be used to describe public health system/ collaborative partnership changes in each ASIST2010 site, and demonstrate that evidence-based strategies can be adapted to other communities and populations. The study will provide a synthesis of information about the public health systems/collaborative partnership approach to inform OWH and policymakers, researchers, practitioners, and the public.
Use of Improved Information Technology and Burden Reduction
The collection of information for this evaluation will be through telephone and in-person interviews, as well as focus groups. Given that the interviews will be conducted via telephone or in-person, the collection of information does not involve the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology. While we will not be using any special information technology procedures to collect information from respondents, the proposed interviews and focus groups will collect only the minimum information necessary for the purposes of the project.
NORC will minimize burden to respondents by providing discussion topics in advance of the call, reducing the burden of the interview and ensuring that the ensuing discussions are focused and require as little time as possible from the respondents. Additionally, NORC will schedule the interviews for a time that is convenient for the respondents, and be accommodating should they need to reschedule.
Efforts to Identify Duplication and Use of Similar Information
OWH commissioned a review of the literature on effective sex- and gender-based models of care, which was completed in January 2007.2 This report concluded that OWH should build on the success of the Centers of Excellence model of care and move toward a gender-based model of health care. It recommended that research be undertaken to explore sex and gender differences and improve data collection. Responding to these recommendations, and building on its previous work to identify successful system change initiatives within health care organizations to improve health outcomes, OWH launched ASIST2010 in 2007. NORC will use the literature as a resource in its assessment of ASIST2010. The literature review is not a duplication of the proposed research effort. The ASIST2010 assessment explores the use of a public health systems/ collaborative partnership approach with a sex and gender focus to improve performance on HP2010 objectives/targets. NORC conducted a literature review to identify duplicative information, and the search did not identify any systematic evaluation of the ASIST2010 program.
Impact on Small Businesses or Other Small Entities
No small businesses will be involved in this study.
Consequences of Collecting the Information Less Frequent Collection
The design of this study requires one data collection activity for five groups of respondents identified in Exhibit 1. The “core” group of grantee team members will be interviewed three times over the project evaluation period. Without collecting this data, OWH will not have access to a comprehensive assessment of the ASIST2010 program, and the overall utility of a sex- and gender-based public health systems collaborative/partnership approach to meeting Healthy People 2010 targets. The federal government will benefit from determining the effectiveness of these approaches to health care.
There are no legal obstacles to reduce the burden of collection.
Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
This request complies with the information collection guidelines of 5 CFR 1320.5. There are no special circumstances.
Comments in Response to the Federal Register Notice/Outside Consultation
A 60-day Federal Register Notice was published in the Federal Register on May 29, 2009, Vol. 74, No. 102 pp. 25750-1 (see Attachment B). There were no public comments.
NORC at the University of Chicago staff consulted on data collection in 2009 include:
Caitlin Oppenheimer, MPH
Associate Director, Public Health Research
(301) 634-9322
Emily Shortridge, PhD, MPH, MPP
Research Scientist, Health Care Research
(301) 634-9343
Rachel Friedman Singer, PhD, MPH, MPA
Research Scientist, Health Care Research
(301) 634-9321
Alycia
Infante, MPA
Principal Research Analyst, Public Health Research
(301) 634-9371
Explanation of any Payment/Gift to Respondents
One honoraria payment in the amount of $30 will be made to each consumer that participates in the focus groups. The use of a monetary incentive has been employed as an effective strategy for increasing response rates. Researchers have found financial incentives to be a motivator for women’s participation in research.3 There will be no payments or gifts to individuals that participate in interviews.
Assurance of Confidentiality Provided to Respondents
Individuals and organizations contacted will be further assured that their replies will be protected under 42 U.S.C. 1306, and 20 CFR 401 and 4225 U.S.C.552a (Privacy Act of 1974). For the in-person and telephone interviews with grantees, representatives from their partner organizations, selected consumers, and representatives from comparison organizations, NORC will be collecting participants’ names, titles, organization, and contact information (phone number, email address, mailing address) in compliance with all aspects of the Privacy Act. Social security numbers will not be collected. NORC will use the contact information to arrange the in-person and telephone interviews. For the consumer focus groups, NORC will collect participants’ names and contact information (phone number, email address, mailing address), in order to schedule a convenient time and place for the focus groups and to remind participants to attend the focus group. Participants will be told the purposes for which the information is collected and that, in accordance with the Privacy Act, any identifiable information about them (e.g., respondent names) will not be used or disclosed for any other purpose. This information will be used solely by NORC to categorize and summarize types of respondents for comparison purposes during the analysis phase of the project. Names will be collected to establish rapport with in-person and telephone interviewees only. Participant names will not be included in any information viewed by OWH or any other HHS officials.
Both interview and focus groups participants will be informed of their rights as study participants.
Participants in the in-person and telephone interviews will be read an informed consent statement prior to participating in the interview (Attachment C). Participants in the consumer focus groups will be asked to complete a written informed consent prior to beginning the focus group (Attachment D). Both informed consent statements indicate that there are no foreseeable risks to participation; participation is completely voluntary; participants have the right to withdraw from the interview at any time; if at any point during the interview the participant withdraws, previous responses will remain part of the record. Participants are free to refrain from answering any questions or commenting on any discussion topics that may arise; whether or not the participant chooses to participate in the interview or focus group, or decides to withdraw at any point, will not affect him/her in any way. The informed consent statements will also ask participants for permission to be audio recorded. Verbal consent will be obtained on the audio recording prior to beginning the interview. An honorarium of up to $30 will be provided to focus group participants for their time, to be paid at the conclusion of the focus group. If a participant withdraws at any point prior to the focus groups’ conclusion, no honorarium will be provided.
Methods will also be taken to protect study data. Each participant will be assigned a unique study identifier to use on copies of the interview protocols and interview/focus group notes. Resulting data from the interviews and focus groups will not identify any person; rather, results will be presented for each ASIST2010 grantee, and across all grantees. Data from the interviews and focus groups will be stored in a password protected database. The crosswalk between participant IDs and participant names will be stored in a password protected Microsoft Excel spreadsheet. NORC will have access to the spreadsheet, and the spreadsheet will be destroyed at the conclusion of the assessment. The briefs and reports will not identify any specific individuals. All potentially identifying information will be destroyed at the study’s conclusion.
As required by Federal law and ethical research standards, all NORC projects involving primary data collection must undergo review by NORC’s Institutional Review Board (IRB). Upon submission of OMB clearance or in the months preceding data collection, NORC will prepare for IRB review.
Justification for Sensitive Questions
The interviews will not include any questions of a sensitive or personal nature. Respondents will be asked to answer from the perspective of their organization about particular aspects of the government programs, as well as the respondents’ opinions of different aspects of ASIST2010. The questions are designed to solicit information solely regarding uses of the initiative in a professional/worksite setting.
Estimates of Annualized Burden Hours (Total Hours & Wages)
12A. Estimated Annualized Burden Hours
In Exhibit 1, we provide estimates of the collection burden on participants from each category of respondent. Data collection activities include (1) interviews with grantee staff; (2) interviews with partner organization staff; (3) interviews with consumers; (4) focus groups with consumers; and (5) interviews with comparison organizations. Draft protocols may be found in Attachments E and F (Round 1 and Round 2 Grantee Staff Interview Protocols), Attachment G (Grantee Site Visit Protocol), Attachment H (Focus Group Discussion Guide), and Attachment I (Comparison Group Interview Protocol). The total cost burden for all primary data collection efforts is $9,558.18.
Exhibit 1. Estimated Burden Hours
Type of Respondent |
Form |
# of Respondents |
No. Responses per Respondent |
Average Burden Per Response (Hours) |
Total Burden Hours |
Grantee Staff |
Attachments E, F |
65 |
3 |
1 |
195 |
Partner Organization Staff (In-person interviews)* |
Attachment G |
52 |
1 |
1 |
52 |
Consumers (In-person interviews)* |
Attachment G |
18 |
1 |
1 |
18 |
Consumers (Focus groups)* |
Attachment H |
40 |
1 |
1.5 |
60 |
Comparison Organization Staff (Telephone Interviews) |
Attachment I |
10 |
1 |
1 |
10 |
TOTAL |
|
167 |
--- |
--- |
335 |
* Data collection activity that will occur during site visits to the thirteen ASIST2010 grantees.
12B. Annualized Cost to Respondents
Exhibit 2. Estimated Burden Cost
Type of Respondent |
Total Burden Hours |
Average Hourly Wage Rate |
Total Hour Cost |
Grantee Staff |
195 |
$31.541 |
$6150.30 |
Partner Organization Staff (In-person interviews)* |
52 |
$31.54 |
$1,640.08 |
Consumers (In-person interviews)* |
18 |
$18.622 |
$670.32 |
Consumers (Focus groups)* |
60 |
$18.62 |
$1,117.20 |
Comparison Organization Staff (Telephone Interviews) |
10 |
$31.54 |
$315.40 |
TOTAL |
335 |
--- |
$9,558.18
|
1 Based on hourly wage for administrators and officials, public administration, “National Compensation Survey: Occupational Wages in the United States, 2005,” U.S. Department of Labor, Bureau of Labor Statistics.
2 Based on hourly wage for all occupations, “National Compensation Survey: Occupational Wages in the United States, 2005,” U.S. Department of Labor, Bureau of Labor Statistics.
* Data collection activity will occur during site visits to the thirteen ASIST2010 grantees.
Estimates of other Total Annual Cost Burden to Respondents or Recordkeepers/Capital Costs
Data collection for this study will not result in any additional capital, start-up, maintenance, or purchase costs to respondents or record keepers. Therefore, there is no burden to respondents other than that discussed in the previous section.
Annualized Cost to Federal Government
All costs for conducting the Evaluation of Advancing System Improvements to Support Targets for Healthy People 2010 (ASIST2010) are included in the contract between the Office on Women’s Health, Department of Health and Human Services, and NORC under contract number HHSP23320082211TC. The total estimated cost is $394,510.00 over a twenty-six month period to conduct the evaluation, analyze and present findings, and write a final report. This is an annualized cost of $182,081.54.
.
Explanation for Program Changes or Adjustments
This is a new collection of data.
Plans for Tabulation and Publication and Project Time Schedule
The data collected will be analyzed and interpreted to produce interim briefings as well as a final study report to the Office on Women’s Health (OWH) at the Department of Health and Human Services. NORC will deliver the final report to OWH in hardcopy and a print-ready electronic format. Publication of findings on the Internet is at OWH’s discretion. The remainder of this section discusses the analytic techniques that will be employed. Information will be collected over a nine month period following OMB approval. Exhibit 3 provides a schedule of data collection, analysis, and reporting following OMB approval.
Exhibit 3. Timetable for Data Collection, Analysis, and Publication
Activity |
Estimated Start Date |
Estimated End Date |
Data collection |
1 month following OMB approval |
9 months following OMB approval |
Data analysis |
2 months following OMB approval |
10 months following OMB approval |
Preliminary briefing and preparation of draft report |
9 months following OMB approval |
10 months following OMB approval |
Final report |
9 months following OMB approval |
10 months following OMB approval |
Final briefing |
10 months following OMB approval |
10 months following OMB approval |
Data analysis will assess whether grantees’ use of a public health systems/collaborative partnership approach that adds a sex and gender focus has helped grantees to meet their Healthy People 2010 targets. In order to answer this overarching question, data analysis will focus on identifying results of the established research questions and sub-questions provided in Exhibit 4.
Exhibit 4: Key Research Questions
1. Have the grantees incorporated structures necessary to positively impact their selected HP 2010 targets? |
|
2. Have the grantees implemented processes that will improve outcomes for their selected targets? |
|
3. What is the grantees experience in carrying out the core activities of ASIST2010? |
|
4. Is there evidence of changes in outcomes as a result of these enhanced structures and processes? |
|
Descriptive statistics and traditional methods of qualitative data analysis - based on the discernment of themes and patterns in the data through an extensive content analysis of the data collected - will be used to analyze data collected through telephone and in-person interviews and through focus groups. For example, descriptive statistics such as frequencies and means will be summarized in tabular format to assess grantees’ organizational demographics (e.g., type of organization, size of collaborative, priority areas of HP2010, partner organizations, activities) and the types of infrastructure changes grantees applied to implement and sustain sex- and gender-based focused care activities; to determine the percentage of grantees that have incorporated various structures in their ASIST2010 projects, such as surveillance or information systems to track clients and to detect sex- and gender-based differences in care and sustainability plans; and to quantify the number of grantees implementing various processes (e.g., proportion of grantees using evidence-based strategies, proportion of grantees employing a sex- and gender-based approach). Qualitative content analysis will be used, for example, to determine the extent to which grantees expanded the reach of their collaborative partnership for ASIST2010, best practices, and their overall engagement in ASIST2010; to assess grantees’ progress in implementing a sustainability plan; and to understanding the processes that grantees are using to meet their HP2010 targets. Common responses will be grouped and categorized for assessment. Analyses will focus on identifying processes that have been particularly helpful in supporting grantees’ activities. NORC will also conduct secondary data analysis of grantee data from the Behavioral Risk Factor Surveillance Survey (BRFSS) to identify changes in outcomes for each ASIST2010 location..
Reason(s) Display of OMB Expiration Date is Inappropriate
OWH does not seek this exemption. All data collection materials will display the OMB expiration details.
Exceptions to Certification for Paperwork Reduction Act Submissions
There are no exceptions to the certification statement.
This study will identify examples of effective strategies and approaches to using public health systems/ collaborative partnerships that add a sex and gender focus to the selected Healthy People 2010 (HP 2010) focus areas. Primary data collection for this study is composed of the following activities:
Semi-structured telephone interviews with ASIST2010 grant directors and relevant staff. Two rounds of telephone interviews will be conducted with the directors and up to four staff members from each of the 13 grantee organizations—the first round will be conducted during the middle of the grant and the second towards the end of the grant. Topics will include experiences with ASIST2010 program activities and progress to date; grantees’ program activities; the impact of using a public health system/ collaborative partnership approach with a sex and gender focus; best practices; consumer satisfaction with the services delivered; and future plans.
Grantee site visits. During site visits, in-person interviews will be conducted with ASIST2010 grant directors, relevant grantee staff, representatives from partner organizations, and consumers. Focus groups with consumers will be conducted in place of interviews with consumers at four of the thirteen grantee sites. Grantees will receive an advance letter, informing them about our plans to conduct a site visit with their staff (Attachment J).
In-person interviews with ASIST2010 staff. NORC will conduct in-person interviews with grantee staff (including directors and up to four staff) at each site, for a total of 65 interviews.
In-person interviews with representatives from ASIST2010 partner organizations. In-person interviews will be conducted with up to 52 representatives from ASIST2010 partner organizations (up to four partners for each of the thirteen grantees) while NORC staff are on-site. The exact number of interviews will vary depending on the ASIST2010 grantee.
Focus groups with consumers. Focus groups will be conducted at four of the ASIST2010 program sites with up to ten participants at each site to learn about consumer satisfaction with program services.
In-person interviews with selected consumer representatives. During site visits, interviews will be conducted with up to four consumers at each of the nine sites not selected for focus groups.
Semi-structured telephone interviews with comparison organizations. NORC will conduct semi-structured telephone interviews with up to ten organizations that are not involved with ASIST2010 to learn about the benefit of designing programs using a public health system/collaborative partnership approach with and without a sex and gender focus. These interviews will be conducted towards the end of the evaluation, after we have preliminary findings from interviews and site visits with ASIST2010 grantees. We will talk with up to five organizations that are using HP 2010 as their objective-setting mechanism, though not using a sex and gender focus, and up to five organizations that provide sex- and gender-focused care but do not use HP 2010 as their guide. The comparative organizations will add valuable information about the combined benefit of designing programs around a sex- or gender-based approach with specific HP 2010 objectives. While the comparison organizations will not technically be control groups, they may allow us to isolate some main effects of each type of approach, along with the interactive effect of having both approaches as the foundation of the ASIST2010 program. Interviews will focus on the advantages and disadvantages of the approaches, consumers’ satisfaction with the program services, overall impact of the approaches, and best practices and lessons learned.
1. Respondent Universe and Sampling Methods
Semi-structured telephone interviews with ASIST2010 grantee directors and relevant staff. This study is surveying the universe of thirteen ASIST2010 grantees. Two rounds of semi-structured telephone interviews will be conducted with the grant director and up to four additional staff from each of the thirteen grantee organizations.
Grantee site visits. We will conduct site visits of all thirteen grantee organizations.
In-person interviews with ASIST2010 staff. Interviews will be conducted with the grant director and up to four additional staff members from each grantee organizations representing a variety of positions and activities. While the list of interviewees will vary across sites, they will most likely include the project director, evaluation director, and various site staff.
In-person interviews with representatives from ASIST2010 partner organizations. Interviews will be conducted with representatives of up to four partner organizations for each of the thirteen grantees. Partner organization representatives will be identified through grantee proposals, progress reports, and our first round of interviews with staff from the grantee organizations.
Consumer focus groups. Focus groups will be conducted at four selected sites based on a review of site activities and consultation with grantee site staff. We will aim to select diverse sites representing various types of organizations (health department, academic institution, hospital, community based organization, foundation); rurality (urban, suburban, rural); socio-demographic characteristics of the target population; and types of activities. NORC will recruit up to 10 individuals for each focus group to be held during selected site visits. Prior experience suggests that a group of this size is appropriate for conducting a focus group that allows for ample exchange of ideas with variation in perspectives while maintaining a controlled discussion that may be skillfully “steered” to the appropriate topics by expert focus group facilitators. NORC will work closely with the selected grantee sites to identify the appropriate recruiting mechanism for a targeted group of respondents. For this evaluation, we plan to select ASIST2010 sites for focus groups that have group activities as part of their programmatic activities which would facilitate focus group recruitment and scheduling (e.g., exercise classes, support groups).
To ensure that we recruit a large enough group, we intend to identify both focus group participants and alternates. While recruitment methods may vary by site, when possible we will first send potential focus group participants an advance letter (Attachment K) signed by the grantee site detailing the relationship between evaluation goals and important outcomes relevant to the targeted population. We will follow up with phone calls, respectfully inquiring about the individual’s willingness to participate and communicating additional information, such as meeting date, time, and place, as appropriate. In other cases, we may work with the grantee site to distribute a flyer to program participants (Attachment L).
In-person interviews with selected consumer representatives. In addition to conducting focus groups with four sites, we will also conduct in-person interviews with consumer representatives from each of the 9 ASIST2010 sites that were not selected for focus groups. Such interviews will help to ensure that we speak with consumers from each of the ASIST2010 sites. We will work with staff from each grantee site to identify up to four consumers from each of the 9 sites that are willing to be interviewed.
Semi-structured telephone interviews with comparison organizations. We will conduct interviews with representatives of up to five organizations that are using Healthy People 2010 as their objective-setting mechanism, though not using a sex- and gender-based framework, and up to five organizations that provide sex- or gender-focused care, but not using Healthy People 2010 objectives as their guide. Potential comparative organizations will be identified in consultation with OWH and through our contacts with Healthy People 2010 and the Racial and Ethnic Approaches to Community Health across the U.S. (REACH U.S.) program. Initially, we will develop a list of approximately 15 organizations using Healthy People 2010 as their objective-setting mechanism and 15 using a sex or gender focus to developing programming. We will narrow the list to approximately five of each type of comparison organization. These organizations will provide a geographically diverse comparison group, addressing several of the Healthy People 2010 focus areas. Comparison organizations will be selected to include organizations addressing similar Healthy People 2010 objectives. We will send potential comparison organizations a letter (Attachment M) requesting their participation in this study.
2. Procedures for the Collection of Information
Exhibit 5 provides an estimated timeline of data collection activities, Funding for ASIST2010 grantees began on September 30, 2007 and ends on September 30, 2010. Evaluation data collection activities will be conducted in the second and third years of the ASIST2010 program.
Exhibit 5: Timeline of Data Collection Activities
Activity |
Estimated Timeline |
Semi-Structured Telephone Interviews with Grantees (Round 1) |
1 month following OMB approval |
Grantee Site Visits |
3 to 7 months following OMB approval |
Interviews with Comparison Organizations |
8 to 9 months following OMB approval |
Semi-Structured Telephone Interviews with Grantees (Round 2) |
9 months following OMB approval |
Semi-Structured Telephone Interviews with Grantees
The purpose of the semi-structured interviews is to expand our understanding of specific ASIST2010 grantee activities and their relationship to HP 2010 objectives. Draft interview protocols for the initial and follow-up structured interviews may be found in Appendices E and F. We plan to conduct two rounds of semi-structured telephone interviews with representatives of the thirteen ASIST2010 sites. Each round of grantee interviews will be conducted with the grant director, at minimum, but may include up to four additional staff members. Additional team members will be encouraged to join, as their perspectives will provide us with a deeper understanding of the grantees’ operations, activities, and strengths and weaknesses. We will contact the grant directors and ask them to choose the appropriate staff to participate in the interview with them.
Conducting two rounds of interviews--in addition to the in-person interviews during the site visits (described below)--with the same groups of grantee representatives will allow us to examine the changes occurring within each of the thirteen grantees’ projects over time. The interview protocols for each round of interviews will address many of the same issues as the initial round of interviews, at a different point in time. The first round of interviews will be conducted approximately 1 month following OMB clearance, at the beginning of the third year of the grantees’ funding. The second round of telephone interviews will be conducted towards the end of the grant period, approximately 8 months following OMB clearance, 2 to 6 months following the site visits.
To limit cost to the government and minimize burden on respondents, interviews will be conducted over the telephone and will last no longer than one hour. Prior to each interview, in addition to pulling together an individually-tailored protocol, the research team will gather and review all information submitted by the grantee, including grant applications and progress reports. The team will also work to gather and review all additional publicly-available information relevant to the respondent grantee site and associated partners.
Interviews will be conducted by at least two team members, one senior staff member to conduct the interview and a junior staff member to take comprehensive notes and identify areas to be covered as the interview progresses. Interviews will be conducted using detailed protocols and note-taking guides that are designed to offer maximum flexibility in gathering input from the diverse group of grantees.
Findings from these interviews will also help us to choose topics for in-depth study during our site visits. For example, if a grantee discusses how they implemented a surveillance system to track consumers, we may focus on the surveillance system during the site visit to the grantee, taking the opportunity to observe the system firsthand, and to explore design, implementation, and evaluation issues.
Following the conclusion of each interview, the study team will produce a clean, electronic copy of interview notes that summarize key findings. Data from the telephone interviews will be compiled in a Microsoft Excel spreadsheet to facilitate a comprehensive analysis of interview findings across grantees.
Grantee site visits
Our third and final point of data collection with the ASIST2010 grantees will occur during grantee site visits. Between the first and second rounds of semi-structured telephone interviews with the 13 ASIST2010 grantees, we plan to site visit them in order to experience their program activities first-hand. The site visit will allow us to assess how their projects have progressed and changed over time.
NORC will prepare an advance letter describing the project, its goals, and the benefits of participation, and send it to ASIST2010 directors (Appendix G). NORC expects to interview a range of people while on site, including ASIST2010 staff members, staff of partner organizations, and consumers of grantee programs.
Interview protocols for staff of grantee organizations will contain a core set of questions which will be asked at each site. Core topics covered in discussions with ASIST2010 staff will include: grantees’ strategies, constraints, successes, data collection and tracking systems, and short- and long-term plans for sustainability.
Draft interview protocols, clearly identified by the type of respondent, may be found in Appendix H. Each protocol is tailored to a particular type of respondent; thus, a grantee site director will be asked a different module of questions than staff from partner organizations or consumers. In addition, we will focus on different objectives and hypotheses (e.g., evaluation plans, outreach) in different sites and perhaps even within a site for different interviewees. To this end, the interview guides are constructed in a modular fashion so that different sets of questions can be combined according to the respondent and topics of interest.
To ensure that we capture each grantee’s unique characteristics and activities, the protocols will also contain site-specific prompts. The protocols will include open-ended questions to encourage informants to share their experiences and concerns. Each guide or set of modules will serve as a checklist, allowing the interviewer to follow the flow of the conversation and explore different avenues of questioning as new issues arise, all the while ensuring that critical topic areas are addressed.
Each site visit will be two to three days in length. As part of the site visits, interviews will be conducted with a range of key informants that represent a variety of positions and activities. Following each site visit, the site visit team will meet with the rest of the project team to conduct a formal debriefing of the findings. This meeting will occur on the next business day that the project team is available. Site visit leads will discuss the findings, any challenges encountered, and unexpected/ anticipated issues. This debriefing will ensure that the entire project team is familiar with the findings from every site visit.
In addition, data from the interviews occurring during the site visits will be compiled in a Microsoft Excel spreadsheet to facilitate a comprehensive analysis of interview findings across grantees.
Assuring that meeting logistics are well coordinated with attention to what will be most convenient for focus group participants is crucial to support adequate participation and, importantly, minimize burden on participants. In all cases, we will work with the selected ASIST2010 grantees to identify the best time and venue for conducting focus groups. To facilitate focus group scheduling, we will do our best to schedule site visits during a time that correlates with another meeting or event that may attract consumers who have been impacted by the program. We will work with grantees to recruit focus group participants through the distribution of a recruitment flyer (Attachment L) to program participants. We will also ensure that participants are able to access the focus group using public transportation and that there is ample parking available at the site. Finally, we will provide participants with an honorarium payment of up to $30 for their time.
As the focus groups with consumers have similar goals to the interviews that we will be conducting with consumers at other sites, the protocols will have some overlap. As with all interview protocols, the focus group protocol will be tailored to specific programs. The draft discussion guide (Appendix K) includes an opening script designed to introduce focus group participants to the overall purpose and structure of the gathering; in addition, there will be a set of opening questions designed to spur an open and informal give-and-take among the group. The guide will help facilitate the meeting and ensure that each of the identified key themes is covered during the meeting.
When possible, we will use three person teams to implement the approved focus group discussion guide. A lead facilitator will be responsible for guiding and moderating the discussion, while the co-facilitator will be responsible for ensuring that all major themes included in the discussion guide are discussed as extensively as possible. A research assistant will take accurate and comprehensive notes about the focus group proceedings and discussion.
We will produce a clean electronic copy of meeting proceedings for our analyses. Notes will be organized around the themes included in the discussion guide and summarized to facilitate major findings. Write-ups will include anecdotal examples of all major points brought up during the focus groups to provide detailed illustration of the ASIST2010 grantees’ impact on the target population to be integrated into the final report. We will not produce verbatim transcripts of focus groups. Provided that we received permissions from every participant, we do plan to audio-record the focus groups to ensure that our notes are accurate and comprehensive. Data from the focus groups will be compiled in a Microsoft Excel spreadsheet to facilitate analysis of the findings.
Interviews with comparison organizations
Interviews with comparison organizations will be conducted towards the end of the evaluation, after we have preliminary findings from interviews and site visits with ASIST2010 grantees.
Once comparison groups are selected, we will contact the directors of selected comparison organizations by email, providing background on ASIST2010, the evaluation, and the goals of our interview with them (see Appendix L for advance letter). When appropriate, we will identify the individual or organization who recommended them as an organization for comparison. We will follow up with each organization by telephone within one week following our initial contact.
NORC will collect qualitative information from comparison (non-grantee) organizations through semi-structured telephone interviews. We will develop semi-structured interview protocols for the comparison groups. These protocols will have elements in common with the protocols for the grantee interviews, but may also contain additional questions. A draft protocol may be found in Appendix M.
The team will work to gather and review all additional publicly-available information relevant to the comparison organization prior to the interview. Interviews will be conducted with at least two NORC team members, one senior staff member to conduct the interview and a junior staff member to take comprehensive notes and identify areas to be covered as the interview progresses. Following the conclusion of each interview, the NORC team will produce a clean, electronic copy of interview notes that summarize key findings. Notes will be developed to facilitate easy incorporation into major evaluation deliverables. Data from the focus groups will be compiled in a Microsoft Excel spreadsheet to facilitate analysis of the findings.
3. Methods to Maximize Response Rates and Deal with Nonresponse
Gaining cooperation and buy-in from grantees are two key objectives of this data collection effort. For this data collection effort, we believe that the specialized nature of the respondent group—grantees that have received funding from the OWH and who are interested in supporting HHS efforts to improve Healthy People 2010 targets—will increase their propensity to respond, and estimate a response rate of 80 percent. It is important to note that the ASIST2010 grantees are aware of this assessment and have agreed to participate in the assessment as part of their grant.
In the case of the in-person consumer interviews and consumer focus groups, NORC may work with the ASIST2010 sites to recruit these individuals. Given the consumers are likely involved with the ASIST2010 program or are receiving its services, it will be effective to collaborate with the ASIST2010 program site to secure the participation of consumers. ASIST2010 sites may suggest appropriate consumers to speak with, and the best times to reach these individuals to schedule the interview/ focus group.
In addition, NORC will use a number of proven methods to maximize participation and cooperation in the study:
The interview protocols will be designed to maximize response rates. The project’s senior staff will ensure that the style of the protocol is inviting and user friendly. The interview and focus group protocols contain questions that are concise.
We will prepare an advance letter (Attachment K) and recruitment flyer for focus group participants (Attachment L).
NORC will include a NORC telephone number and email address within the cover letter, in case participants have questions about the study or their participation.
NORC will follow up with individuals by telephone to encourage participation in the interviews.
While we expect a few hard refusals to the interviews and focus groups, we have found that these techniques are highly effective ways to increase response rates, particularly in cases where the overall sample size is small enough to provide a “personalized” or high-touch level of follow-up.
4. Tests of Procedures or Methods to be Undertaken
No pilot testing of data collection instruments will be conducted.
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or
Analyzing Data
The following individuals contributed to the questionnaire and study design and will be involved in the interpretation and analysis of findings:
Caitlin Oppenheimer, MPH
Associate Director, Public Health Research
NORC at the University of Chicago
(301) 634-9322
Emily Shortridge, PhD, MPH, MPP
Research Scientist, Health Care Research
NORC at the University of Chicago
(301) 634-9343
Rachel Friedman Singer, PhD, MPH, MPA
Research Scientist, Health Care Research
NORC at the University of Chicago
(301) 634-9321
Alycia Infante, MPA
Principal Research Analyst, Public Health Research
NORC at the University of Chicago
(301) 634-9371
The government project officer for this study is:
Barbara
F. James, MPH
Director, Division of Program Coordination
DHHS Office on Women's Health
(301) 443-1402
Attachments for Supporting Statement, Parts A and B
Attachment A: OWH Authorizing Legislation
Attachment B: Federal Register Notice
Attachment E: Round 1 Grantee Interview Protocol
Attachment F: Round 2 Grantee Interview Protocol
Attachment G: Grantee Site Visit Protocol
Attachment H: Focus Group Discussion Guide
Attachment I: Comparison Organization Interview Protocol
Attachment J: Site Visit Advance Letter for Grantees
Attachment K: Focus Group Advance Letter
Attachment L: Focus Group Flyer
Attachment M: Advance Letter for Interviews with Comparison Organizations
1 Brittle C, Bird C. 2007. Literature Review on Effective Sex- and Gender-Based Systems/ Methods of Care. Produced for the Office on Women’s Health, U.S. Department of Health and Human Services. Available at: http://womenshealth.gov/owh/multidisciplinary/reports/GenderBasedMedicine/FinalOWHReport.pdf.
2 Brittle C, Bird C. 2007. Literature Review on Effective Sex- and Gender-Based Systems/ Methods of Care. Produced for the Office on Women’s Health, U.S. Department of Health and Human Services. Available at: http://womenshealth.gov/owh/multidisciplinary/reports/GenderBasedMedicine/FinalOWHReport.pdf.
3 Brown BA, Long HL, Gould H, Weitz T, Milliken N. 2000. A Conceptual Model for the Recruitment of Diverse Women into Research Studies. Journal of Women’s Health & Gender-Based Medicine 9(6):625-632.
| File Type | application/msword |
| File Title | How to Write and Submit |
| Author | CMS |
| Last Modified By | DHHS |
| File Modified | 2009-08-07 |
| File Created | 2009-08-07 |