20078 ID_Girls at Greater Risk Supporting Statement OMB 9_27_13 v2

Supporting Statement for Request for Clearance:

GIRLS at GREATER RISK FOR JUVENILE DELINQUENCY AND HIV PREVENTION PROGRAM

Contact Information:

Aleisha Langhorne, MPH, MHSA

Health Scientist

Office on Women's Health

Office of Public Health Science

U.S. Department of Health and Human Services

200 Independence Avenue, SW, Room 733E

Washington, DC 20201

202-401-8325

202-401-4005 fax

Aleisha.Langhorne@hhs.gov

August 22, 2013

SUPPORTING STATEMENT

GIRLS AT GREATER RISK FOR JUVENILE DELINQUENCY AND HIV

PREVENTION PROGRAM

A. JUSTIFICATION

A.1 Need and Legal Basis

In 1998, as the result of the HIV/AIDS state of emergency declared by African American community leaders and supported by the Congressional Black Caucus (CBC), Congress funded an initiative to address this crisis through increased funding and outreach. This initiative would eventually become known as the Minority AIDS Initiative (MAI). These funds targeted HIV/AIDS programs that directly benefit racial and ethnic minority communities. The MAI is not a part of the Ryan White Care Act authorizing legislation, but provides directed resources to some CARE Act programs, as it does to other Public Health Service HIV/AIDS programs.

The MAI takes a multi-faceted approach that focuses simultaneously on HIV prevention, care, treatment and research. Further, within these broad categories of funding, MAI funds direct services, technical assistance, training and capacity-building, and evaluation.¹ Legislation that gave birth to MAI came as a result of the Omnibus Consolidated and Emergency Supplemental Appropriations Act of 1999, PL 105-277, October 21, 1998 and was initially referred to as the “CBC” initiative. A copy of the legislation may be found at the following website: http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=105_cong_public_laws&docid=f:publ277.105.pdf

The current data collection is being requested for an evaluation of the Girls at Greater Risk for Juvenile Delinquency and HIV Prevention Program (Girls at Greater Risk). This is a new data collection requesting OMB approval. Due to the concern that rates of HIV infection among adolescents of color are increasing, the Department of Health and Human Services (HHS), Office on Women’s Health (OWH) is seeking to evaluate its funded HIV prevention programs which target girls between the ages of 9 and 17 years in order to identify best practices and the most effective gender-responsive approaches to HIV/AIDS prevention. (See Appendices A.1 to A.3 for background information on OWH, its programs and girls/women and HIV/AIDS).

The primary goal of OWH’s Girls at Greater Risk program is to increase knowledge about HIV and other sexually transmitted infections (STIs) and individual protective factors against juvenile delinquency among girls between the ages of 9 and 17 years using gender responsive prevention strategies. The program targets those behaviors and attitudes that have been found to promote both juvenile delinquency and sexually risky behavior. The program also focuses on building self-esteem and overall sense of well-being and mental health and on educating girls on how to cope with adverse circumstances and situations. Lastly, the program hopes to build the capacity of community organizations to service the needs of female adolescents and youth.

OWH will use the evaluation findings in making programmatic and funding decisions about their Girls at Greater Risk program. Evaluation findings will also be used to identify best practices and lessons learned that could be transferred to other organizations and communities attempting to replicate services for at risk girls and female adolescents. Furthermore, because this program was developed as part of the national effort to eliminate health disparities, evaluation findings will help OWH understand how the program is contributing to closing the health disparities gap and to improving care for underserved populations of women and girls across the country.

A.2 Purpose and Use of Information

The purpose of this information collection is to gather data from grantees participating in OWH’s Girls at Greater Risk program related to program efficacy. Information obtained from this data collection will be used to assist in measuring the effectiveness of OWH’s efforts to reduce risk of acquiring HIV among girls and female adolescents and to increase HIV prevention knowledge among girls. All data collection forms and activities are designed within the parameters of a four-year evaluation of the Girls at Greater Risk Program.

This evaluation will enhance OWH’s capacity to identify, support, and create effective HIV prevention programs for girls and adolescent females. This data collection will also help to improve OWH’s knowledge of gender responsive methods to reduce risk of contracting HIV and increase HIV prevention knowledge among adolescent females. In addition to informing the development of current OWH HIV prevention programs, the evaluation results will also aid in the planning and development of future OWH and other public and private sector HIV prevention programs.

Failure to collect this information will have negative consequences on HIV prevention efforts among girls and adolescent females, especially among females of color. These data permit OWH to enhance its knowledge of effective program planning, development and delivery and to continue to work toward eliminating racial and ethnic disparities among women. The data will be used to increase OWH’s knowledge base of gender-centered intervention models, provide guidance to funded programs and develop best practices for HIV prevention programs funded in the future.

Overall, the evaluation of OWH’s Girls at Greater Risk program will assess the effectiveness of this program in delivering gender-centered intervention models. Data will be collected through surveys, focus groups and interviews. There are six potential respondent types involved in the HIV prevention programs. They are participants (girls and adolescent females), parents, program partners, program directors, program staff and community residents who participate in grantee community-wide events. This type of evaluation methodology assesses program effectiveness over a 21 month period and in a manner which is conclusive and efficient.

The DHHS-Office on Women's Health intends to use the evaluation results of the HIV/AIDS programs to address the PART deficiencies indicated by the Office of Management and Budget in 2004. The evaluation will address several of the objectives for program management, strategic planning and program results. Additionally, the evaluation results are critical to measuring the efficacy of the use of government funds.

A.3 Use of Improved Information Technology and Burden Reduction

Program staff will be collecting data on an ongoing basis from program participants and will be reporting information to OWH quarterly. To reduce respondent burden, OWH will create a web interface allowing grantees to input data directly into a data repository. All grantees have Internet access and will easily be able to input required data into this system. OWH and Global Evaluation & Applied Research Solutions (GEARS) Inc., the contractor engaged to conduct the evaluation, will have 24-hour access to data that is uploaded by grantees on a quarterly basis. GEARS will use the data to compile quarterly analyses to generate its quarterly report to OWH.

The process evaluation assessments will be administered by GEARS staff via personal and telephone interview with the program directors, staff and program partners of the funded grantees. (See Appendices B.1 to B.3.) The process evaluation questions require answers in a narrative format in order to obtain a comprehensive and coherent response. Conducting either an in-person or telephone interview requires less of the respondents’ time than composing and typing narrative for an online process interview. Responses recorded by GEARS staff will be entered into a qualitative software program for data analysis and retrieval.

A.4 Efforts to Identify Duplication and Use of Similar Information

No effort to collect similar data is being conducted within the agency. Additionally, no data collection efforts outside the agency have been made to collect this data. The respondents are participants in a new OWH program and the data are specific to the evaluation of this program.

A.5 Impact on Small Businesses or Other Small Entities

This data collection involves the collection of information from small businesses or other small entities. We have created the data collection methodology to include the minimal amount of information required to effectively evaluate the program. Additionally, as much as possible, we have attempted to collect data that respondents will currently need to maintain and locally evaluate their programs. Therefore, small business and other small entity respondents (e.g. community based organizations) will primarily submit information that they needed to collect for their own purposes. Grantee proposals were reviewed to ascertain what program participant data grantees planned to normally collect as part of their program implementation. This data, along with data needed for the proposed data collection, was integrated into a data collection instrument. This data collection instrument was designed to request and utilize information that programs (i.e., grantees) collect as part of their required in-house evaluation activities and that imposes the minimal amount of burden as possible. The data collection requirements and survey questions have been held to the absolute minimum required for the intended use of the data. Each grantee has reviewed and approved the proposed data collection form regarding information on program participants.

Additionally, during the grantee orientation meeting and in a subsequent two-day meeting, the evaluation team met with all grantees and discussed the least burdensome data collection mechanism and frequency of reporting. Grantees supported submitting data through the Internet on a quarterly basis. The grantees supported development of forms that maintain all of the information they currently collect and submission of data on the Internet. Directors of the awarded grants also indicated the positive impact that the evaluation results will have on their program effectiveness and opportunities for future funding.

A.6 Consequences of Collecting the Information Less Frequent Collection

This is a one-time data collection effort with six respondent types program directors, program staff, program partners, program participants, parents of program participants and community residents. Approval is sought for four years.

We are requesting that grantees report information quarterly. We will collect two kinds of information from grantee staff (i.e., program directors and program staff). The first type of information is information that they collect from program participants that will be shared with OWH and the second type of information is information collected from grantee staff regarding their program and related activities. Information collected about program participants from grantee staff will be collected on a quarterly basis. This information is collected at this frequency in order to allow grantee staff time to verify and enter collected data. A less frequent data collection would increase the probability of errors.

Information collected by program staff from program participants will be collected at pre-test (the beginning of the nine-month core intervention); post-test (at the conclusion of a nine-month core intervention); six months after post-test; and 12 months after post-test. For each grantee, GEARS will hold one focus group with a subsample of participants and one with a subsample of parents. Community residents will provide feedback on community events in which they participate.

Information collected about the program from grantee staff will be collected twice a year from program directors and program staff. GEARS will collect program information from grantee staff. This frequency is requested in order to assess program changes during the course of the grant. A less frequent data collection increases the probability that grantee staff may not recall as thoroughly programmatic information critical to the evaluation.

Information from program partners will be collected once a year. GEARS will collect program related information from program partners. This frequency is requested in order to assess program and partnership changes during the course of the grant. A less frequent data collection increases the probability that program partners may not completely recall information critical to the evaluation.

If this information collection is not conducted, OWH’s ability to accurately measure and evaluate the impact of this program against its stated objectives will be negatively affected. Failure to include these data collection activities as part of the overall evaluation design will limit the validity of the results and negatively impact the health of at risk girls and female adolescents. There are no legal obstacles to reduce respondent burden.

A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

The proposed evaluation fully complies with all guidelines of 5 CFR 1320.5 (d) (2). The information collection will not be conducted in a manner:

• Requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;

• Requiring respondents to submit more than an original and two copies of any document;

• Requiring respondents to report more often than quarterly;

• Requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records, for more than three years;

• In connection with a statistical survey that is not designed to produce valid and reliable results that can be generalized to the universe of the study;

• Requiring the use of a statistical data classification that has not been reviewed and approved by OMB;

• That includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or,

• Requiring respondents to submit proprietary trade secrets, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.

A.8 Comments in Response to the Federal Register Notice/Consultation

The data collection notice for the evaluation of the Girls at Greater Risk Program was published in the Federal Register, volume 78, number 144, page 45200 on July 26, 2013. A copy of the Federal Register notice is included as Appendix C. There were no comments received from the public regarding this data collection.

The DHHS/OWH Project Officer for this data collection is Aleisha Langhorne. Additionally, OWH engaged the consulting firm Global Evaluation & Applied Research Solutions (GEARS), Inc to assist in the development of the survey instruments and evaluation methodology for this evaluation. GEARS is experienced in managing and conducting evaluations and provided expertise on issues including the availability of data, frequency of collection, clarity of instructions, record keeping, confidentiality, disclosure of data, reporting format, and necessary data elements. Also, in 2007 GEARS completed the OWH evaluation of its Rural South, Incarcerated/Newly Released and Mentoring Partnership programs. This evaluation was approved by OMB. Also, GEARS is currently conducting an OMB approved data collection for the Office on Women’s Health for the “Evaluation of the HIV Prevention Program for Young Women attending Minority Institutions.”

A.9 Explanation of Any Payment or Gift to Respondents

There will be no payment, gift, or reimbursement to respondents for time spent.

A.10 Assurance of Confidentiality Provided to Respondents

The evaluation contractor, GEARS, Inc will not collect identifying, personal data on program participants and all information collected from grantees on program participants (girls and adolescent females) is de-identified. Grantee program staff responsible for survey administrator will inform all potential respondents (program participants and their parents or legal guardians) of the purpose of the survey, how the information collected will be used, and that no personal identifiers will be associated with their responses will be shared with the national evaluation contractor. The Prevention Education Questionnaires for Girls, (Pre-test, Post-test, Follow-up; See Appendices D.1 to D.6) administered to program participants by grantees will not collect on the survey participants’ names or other identifiers that will allow survey responses to be linked to individual participants. Girls will use a formula to develop their unique identifier (See Appendix E). Information will be kept private to the extent possible by law. However, grantee local evaluators will be able to link unique identifiers to girl’s responses. This is being done in the event that girls report experiences such as rape or severe depression that appropriate responses (e.g. mandatory reporting, seeking appropriate resources, etc.) can be implemented.

Data collection concerning minors requires human subjects approval by an Institutional Review Board (IRB). Each grantee will obtain IRB approval for data collection for their local evaluation, which includes survey data for the current evaluation. The evaluation activities will be conducted after approval has been obtained by an Institutional Review Board (IRB). Each institution will submit a protocol to their IRB and if a grantee has no IRB, GEARS will submit a comprehensive protocol to its IRB. No data will be collected without IRB approval. Parental Consent forms approved by the grantees IRB and assent forms for youth will be obtained before program participants take any questionnaire. This consent form will assure grantees and program participants that their services will not be adversely impacted by their decision not to participate in the survey. It also informs parents and girls about the limits of confidentiality and under what circumstances confidentiality is broken and what happens in those circumstances.

Program participants and parents/legal guardians of participants will be randomly selected to participate in focus groups. Grantees (programs) will provide GEARS with identification numbers of program participants. Using random selection (e.g. use random numbers table to select starting point and then select every nth participant) GEARS will select program participants for the focus group. GEARS will provide the identification numbers of the selected participants to the grantee and the grantee will contact the participant to determine their desire to participate. A second random selection process will be undertaken where the selected participants’ parent/legal guardian will be identified and asked whether they wish to participate in the focus group. Program participants and the parents/legal guardians who are randomly selected to participate in focus groups about the program will be known to the researchers. However, no personal information will be asked during the focus group. Participants and parents/legal guardian’s opinions about the effectiveness of the program will be collected. Focus group participants will be asked to give their written consent. Girls will require consent from their parents in addition to providing their assent.

Consent procedures will also be used by GEARS in collecting information from program directors, program staff, and program partners. All consent procedures and forms used for these respondents will be approved by GEARS IRB.

The questionnaires will be administered to program participants by grantee staff. Survey responses will be coded, entered into a database and electronically submitted to GEARS to use only for data analytic and evaluation purposes.

A.11 Justification of Sensitive Questions

This evaluation asks sensitive questions. These questions represent standard techniques used in public health practice in assessing the burden of HIV/AIDS on communities and populations and in assessing the public’s level of knowledge and risk behaviors. Moreover, all questionnaires used in the evaluation would have been reviewed by an Institutional Review Board to ensure that respondents’ rights are protected.

Sensitive questions are asked on the three versions of the Prevention Education Questionnaire for Girls, (i.e. the pre-test, post-test, and follow-up; see Appendices D.1 to D.6). For each version there is a questionnaire for girls 9 to 11 years and a version for adolescent females 12 to 17 years. The sensitive questions asked on the pre-test version of the questionnaire are also asked on the post-test and follow-up versions of the questionnaire. The types of questions asked are also asked by the grantees in their local evaluation. Grantees will also use the data from the sensitive questions asked in their local evaluations. The sensitive information has been standardized across all grantees for use in this evaluation. Grantees will advise their program participants that their participation and responses to questions are voluntary.

The three versions of the Prevention Education Questionnaire for Girls (PEQG) collect information from grantees about their program participants’ social and demographic information, self-perceptions, delinquent behaviors, sexual practices, dating attitudes and experiences, exposure to violence, school attendance, sexual orientation, substance use and history of sexual assault victimization. Questions regarding sexual practices, dating attitudes and experiences, sexual orientation, and sexual assault are only asked of adolescent females age 12 to 17 years. Much of this information is sensitive; however, program participants provide much of the information requested in the course of the program’s local evaluation data collection. We are collecting data that grantees have indicated is needed in the assessment of their programs. The GEARS evaluation team does not receive any personal identifying information about program participants from the grantees. Each program participant has a unique identifier only known to the participant and grantee data collector. Participants are provided a formula to create this unique identifier so that it can be used during pre-test, post-test, and follow-up. (See Appendix E for the unique identifier instructions.) Grantees participating in the demonstration programs will forward questionnaire information to the GEARS evaluators. Collectively, the sensitive information asked on the PEQG provides a profile of the clients served by OWH funded grantees with respect to HIV risk behaviors and status and on key indicators that have been empirically associated with HIV/AIDS risk for girls and adolescent females. This profile can be linked in data analysis to evaluation outcomes, such as risk behavior practices, in order to provide a better understanding between the association between participant socio-demographic information, the intervention and participant outcomes. For HIV prevention program planning this information is critical.

In sum, the sensitive information requested provides an opportunity to examine the effectiveness of grantees’ programs in reducing those personal, social and behavioral risks associated with HIV/AIDS for program participants. Including these factors among the evaluation variables is critical to determining the degree to which OWH programs are effective.

A.12 Estimates of Annualized Hour and Cost Burden

This evaluation is a one-time effort conducted for three years with an estimated 4,401 annual burden hours. The evaluation will be targeted to approximately 1,450 respondents among six respondent types: program participants, parents of program participants, program directors, program staff, program partners and community residents who are not enrolled in the program. Exhibit A.1 presents the hourly burden breakdown which was used to derive the total burden time. Exhibit A.2 presents the annualized hourly costs for respondents.

The burden to respondents who participate in the evaluation will be in terms of their time.

Exhibit A-1 Estimated Annualized Burden Hours

Type of Respondent	Form Name	Number of Respondents	No. Responses per Respondent	Average Burden per Response (in hours )	Total Burden Hours
Program participants (girls and female adolescents)	Prevention Education Questionnaire: two administrations in 12 months).	750	2	2	3,180
	Focus groups	120	1	90/60
Parents of program participants	Focus groups	120	1	90/60	180
Program Staff	Process Evaluation Interview	10	2	45/60	15
Program Directors	Process Evaluation Interview: Program Directors	10	2	90/60	30
Program Partners	Process Interviews	60	1	45/60	225
	Focus Group	120	1	90/60
Community residents	Workshop evaluation	250	1	5/60	21
Total		1,450			3,651

Exhibit A-2 Estimated Cost Burden

Type of Respondent	Total Burden Hours	Hourly Wage Rate	Total Respondent Costs
Program participants	3,180	$7.25	$23,055
Parents of program participants	180	$14.00	$2,520
Program Directors	30	$25.00	$750
Program Staff	15	$14.00	$210
Program Partners	225	$14.00	$3,150
Community residents	21	$14.00	$294
Total			$29,979

Program participants will complete two questionnaires as part of their participation in the program per year. During the first 12 months participants will complete a pre-test and post-test questionnaire and during the second 12 months they will complete follow-up questionnaires (one at six months after the post-test and one at 12 months after the post-test). These questionnaires will be administered in two different administrations and in total take two hours. A subset of program participants and a subset of parents will participate in a 90 minute focus group. Participant and parent focus groups will be held separately at each grantee site. Program directors and staff and program partners will participate in a process evaluation interview. The interview for directors will take 90 minutes, for program staff 45 minutes and for program partners 45 minutes. Program partners will also participate in a 90 minute focus group. Community residents will take a brief, five-item survey about their experience at the community event (See Appendix F for Community Event Form).

A.13 Estimates of other Total Annual Cost Burden to Respondents or Recordkeepers/Capital Costs

There are no additional respondent costs associated with start-up or capital investments. Additionally, there are no operational, maintenance or equipments respondent costs associated with continued participation in the evaluation.

A.14 Annualized Cost to the Federal Government

The evaluation will be conducted for four years. The overall cost to implement the evaluation is associated with labor required to conduct the following activities: develop evaluation design and methodology; develop data collection forms; design and develop electronic data storage systems; manage data collection activities; develop quarterly reports; conduct and report site visits to funded contractors; develop the evaluation methodology and analysis plan; train evaluation staff; administer interviews; perform data entry; ensure accurate data maintained in data storage systems; and analyze and report evaluation results. Exhibit A-3 presents the cost breakdown by major budget category.

Exhibit A-3 Cost of the Proposed Study

Activity	Cost
Personnel Costs (GEARS and federal employee)	$186,814.65
Other costs (facilities, travel, postage, copying supplies, conf. calls, site visit)	$120,491.11
Total	$307,305.76

Total annualized costs to conduct this evaluation are $307,305.76.

A.15 Explanation for Program Changes or Adjustments

There are no changes in burden. This is a request for an extension.

A.16 Plans for Tabulation and Publication and Project Time Schedule

Exhibit A-4 Project Time Schedule

Activity	Time Period
Federal Register Notice and OMB Clearance	March-August 2010
Administrator/Data Collection Training	April-June, 2010; August 2010
On going data collection activities	September 2010– Dec 2013
Analysis (ongoing data analysis)	October 2010-January 2013
Reporting: Interim Brief and Preliminary Analysis	August 2011; August 2012
Reporting: Evaluation Report & Executive Summary	March 2013

Publication

Evaluation findings will be summarized in a comprehensive Evaluation Report and Executive Summary developed by GEARS for OWH. The findings from this evaluation will be shared with professionals working with HIV-infected women at regional and national conferences.

Analysis Plan

Both qualitative and quantitative data will be collected for this evaluation. Data analysis will be supervised by Deborah Brome, Ph.D., Project Evaluation and Data Manager, in consultation with Michael Milburn, Ph.D., Project Statistician. Data entry, file organization and data access and management will be supervised by Dr. Deborah Brome.

A. Qualitative Data Analysis. Qualitative data will consist of structured individually administered interviews. These data will be analyzed using the basic strategies and principles espoused from grounded theory and the interpretative process. From Grounded Theory, the data analytic strategy will focus on the systematic examination of data for the purposes of explicating the inter-relationships between concepts that assist with the testing of hypotheses. The interpretative process will provide a context for understanding the data gathered, especially as it pertains to culturally significant processes.

Each structured interview (process evaluation interviews) will be coded for themes relating to our project objectives. Coding organizes and identifies issues and themes of relevance in the text. Coded data may be descriptive, interpretive or identify patterns. Codes are used to provide labels or tags assigning meaning to the descriptive information provided by the interviews. GEARS will develop the coding scheme for the process interview protocols.

Following the initial coding staff will meet to review coding and present the analysis to the project team. Codes will be modified through a consensus among the team members. Inter-rater reliability of the coding scheme will be established among interview coders,

Following manual coding, data will be entered into a computer using NVivo, a qualitative data analytic software program. NVivo allows the researcher to code, search, sort, retrieve and find relationships among text. The initial codebook will be entered into the program with any notes or memos by project staff taken in the initial review of the data. Subsequent to entering the interview data, an initial search for several single codes will be performed and adjustments made to the codebook and coding as necessary.

A final analytic step goes beyond classification of the data and explores whether or not linkages exist between/among particular categories. At the descriptive level, analysis involves seeing patterns. At the theoretical level, it involves thinking about why things happen. NVivo will be used in this step.

In addition, structured interview data is amenable to analysis through nonparametric tests using SPSS. Where appropriate qualitative data will be coded and entered into an SPSS program for analysis. For example, through SPSS we can analyze the number and types of prevention activities offered by grantees.

2. Quantitative Data Analysis. Quantitative data will consist of measures of prevention knowledge and attitudes and an appraisal of risk reduction behaviors.  Once the data have been entered and cleaned, statistics such as frequencies, percentages, means, and standard deviations, as well as reliability statistics will be calculated.  The basic research design utilizing quantitative data is a mixed design with between-groups factors (e.g., program, older versus younger adolescents, peer led versus adult led interventions, etc.) and within-group factors (time) repeated measures design, with four measurements taken at pre-test, post-test and two follow-ups for each cohort of participants.  There will be 10 programs that will each serve two cohorts of 75 girls, ages 9 to 17.

 

There are three primary dependent variables, HIV risk reduction, juvenile delinquency risk reduction, and HIV and STI knowledge.  The reliability of these measures will be assessed as with the utilization questionnaire, using factor analysis and reliability analysis.  Additionally, there are a group of questions that assess gender specific risk factors.  These questions have been generated from the curricula used by the minority institutions grantee organizations and in collaboration with the principal investigators of the minority institution grantees Girls at Greater Risk HIV prevention programs.  As with all the risk reduction and HIV knowledge questions, factor analysis and reliability coefficients will be computed for these scales.  In addition, a structural equation model will be estimated that specifies and separately estimates both the amount of change over time in the measures and the reliability of the measures.  Ordinary test-retest analysis confounds reliability and change over time. The Wheaton model for the analysis of panel data will be employed to separate these different components. In order to ensure correct standard errors, a complex sampling design will be specified, so that participants will be nested within their individual program.

 

After the reliability of our instruments has been determined, the primary analysis we plan is a mixed model design, estimated with the General Linear Model program in SPSS.  This model will include one within-subjects factor of time and two between-subjects factors of program (10 levels) and developmental stage (early, middle, or late adolescence) with participants’ scores nested within their program.  Background variables such as age and grade will be entered as covariates. While there will be main effects between programs over time, the time by program interaction will provide an assessment of difference between programs.  Doing a power analysis (Cohen, 1977) using tables for interactions in repeated measures analysis of variance (Potvin & Schutz, 2000), we can make a judgment concerning the necessary sample size.  Hypothesizing a medium effect (ES= .50), the time (3 df) by group (9 df) interaction has 27 degrees of freedom. So to obtain power=.80 at p=.05, approximately 37 subjects per group are needed (Potvin & Schutz, Table 1, page 352).²  The proposed available sample size thus ensures quite adequate statistical power.

 

Assessing differences between grantees programs allows us to identify those that are particularly effective; process analysis will enable us to determine what aspects of their programs are particularly effective.  OWH can make use of important aspects of their programs as they consider future initiatives.

A.17 Reason(s) Display of OMB Expiration Date is Inappropriate

OMB expiration dates will be displayed on all materials.

A.18 Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification statement identified in item 19 “Certification for Paperwork Reduction Act Submissions,” of OMB Form 83-I.

B. Collection of Information Employing Statistical Methods.

B.1 Respondent Universe and Sampling Methods

There are six respondent types participating in this evaluation project. They are program directors, program staff, program partners, program participants, parents of program participants and community residents. No sampling methods are used for any respondent type. Instead a convenience sample will be used since data is being collected from all grantee staff and program participants. Moreover, program directors, program staff, and program partners of funded organizations are predetermined based on awards made in 2009 by DHHS/OWH. Therefore, the entire universe of respondents for these groups consists of the program personnel and partners of organizations that received grants for this project.

The respondent universe for program participants consists of those girls and female adolescents between the ages of nine and 17 years who are recruited for and participate in the organization’s Girls at Greater Risk program. It is expected that annually 750 girls and adolescent female participants will be recruited among the 10 grantee organizations. It is expected that 80 to 100 percent of program participants will participate in the pre and post-test data collection. This estimate is provided given the requirement of programs funded by the OWH Girls at Greater Risk program to retain 80 percent of their program participants. Grantees will use a variety of strategies to maintain program participation, including individual one to one meetings between participants and staff; maintaining contact with the participants’ parents or guardians; and holding community events and activities.

B.2 Procedures for the Collection of Information

There are several data collection activities included in this project. Appendix G provides a description of the information collected, rationale and sources/references for each data collection used in this project. The following presents data collection procedures for the evaluation project:

1. Verify IRB approval from all institutions. GEARS will collect IRB approval letters from all institutions participating in the evaluation.

2. Conduct site visits to all grantees and provide technical assistance on how to obtain consent/assent, administer questionnaires and how to store data.

3. Obtain OMB clearance.

4. Finalize all forms with the OMB clearance number printed on forms.

5. The day OMB clearance is received, GEARS will send an email to each program to inform them that: 1) OMB clearance has been obtained and 2) that they will be receiving within the next three days via email data collection forms and a database with instructions on how to input data. They will also be informed of the dates on which a webinar will be given on how to use the database.

6. Two weeks after receiving OMB clearance, grantees will participate in a webinar conducted on how to enter data into the data repository. This webinar will be repeated once a month for two months. In addition, ongoing technical assistance will be provided for grantee staff who are entering data into the database.

7. Two weeks after receiving OMB clearance, GEARS will begin training sessions for GEARS staff on details of the administration of the process evaluation interviews. This two-hour training will review the purpose of the process evaluation and how the interview fits within the evaluation framework. Each process interview question will be discussed and questions answered. This training will be led by GEARS’ evaluation team Senior Evaluator. GEARS staff is trained in interviewing administration and have conducted interviews as part of GEARS’ research and organizational development activities.

8. One month after OMB clearance begin conducting monthly conference call in order to monitor and answer questions about data collection procedures.

9. Approximately one to two months after OMB clearance, GEARS will attend the OWH Grantee Meeting and will provide technical assistance to grantee staff on data collection from participants and on data entry. Emphasis will be placed on the consenting process that has been approved by their IRB’s and utilizing the script developed for creating the unique identifier. (Appendix E)

10. On a quarterly basis throughout the project period, respondents (program staff) will enter the data into Excel spreadsheet and transmit results to GEARS. Program staff will also use submitted data as part of their local evaluations.

Submission of Data

Data collected for this evaluation are designed to fit within existing program data collection activities as required by the grant. Management Information Systems (MIS) information as well as data markers required for the evaluation will be entered into a user friendly Internet based, data repository on a quarterly basis. GEARS Senior Evaluator will make a site visit of all programs and conduct focus groups with a subsample of program participants and with a subsample of parents. During these site visits the GEARS Senior Evaluator will also provide technical assistance regarding data collection and submission.

Process evaluation data will be collected twice a year. These data will be collected by telephone interview and during site visits made for data collection purposes (i.e. conducting focus groups). GEARS staff will ask the respondent permission to audiotape the process evaluation interview. Audio-taping is requested in order to ensure that the detailed information provided by the respondent in this interview is accurately captured in the evaluation data. Specifically, GEARS staff will check the accuracy of the recorded interview responses against the audio-taped interview. All audio-tapes will be kept in a locked filed in GEARS offices and will be destroyed at the conclusion of the evaluation project, when all data and data coding have been finalized.

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

The questionnaire administered to program participants 12 months after the post-test is the only data collection that is not being collected as part of program activities. In order to increase response rates for this data collection OWH grantees will utilize best practices for working with youth which require that the program continue to maintain contact with girls and adolescent females when they are not actively participating in the program. These methods include inviting participants to community outreach events, reunion/alumni parties and holiday events; providing incentives; and keeping in touch through program newsletters, birthday cards, telephone calls (cell phones, texting reminders); and the program’s Facebook page (or other social networking tools).

One hundred percent participation is expected from OWH grantees in collecting process evaluation data. This data will be collected during telephone interviews and as part of the data collection site visit.

B.4 Test of Procedures or Methods to be Undertaken

The majority of the Instruments and items chosen for this evaluation have been selected from standardized instruments previously used or recommended by federal organizations, previously OMB approved evaluations, research groups, or academic institutions with either community-based populations or organizations and tested for their cultural and linguistic appropriateness.

Additionally, two sets of respondents (program directors and program staff) had opportunities to input on development of the instruments and submit copies of instruments they currently use in their local evaluations to guide development of instruments used in this project. Every effort was made to ensure that data collected for OWH was also data needed by grantees for their local evaluation.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Program Development Contact

Aleisha Langhorne, MPH, MHSA

Health Scientist

U.S. Department of Health and Human Services

Office on Women's Health

202-401-8325

Aleisha.Langhorne@hhs.gov

Data Collection/Analysis and Statistical Contact

Deborah Brome, Ph.D.

Vice President and Director, Evaluation & Applied Research

Global Evaluation & Applied Research Solutions (GEARS Inc.)

301-429-5982

dbrome@getingears.com

Michael Milburn, Ph.D.

Professor of Psychology

University of Massachusetts, Boston

100 Morrissey Boulevard

Boston, Massachusetts

617-287-6386

Michael.Milburn@getingears.com

 

1 Aragon, R. & Kates, J. Minority AIDS Initiative Policy Brief, Kaiser Foundation, June 2004

2 Potvin, P. J., & Schutz, R. W. (2000). Statistical power for the two-factor repeated measures ANOVA. Behaivor Research Methods, Instruments, & Computers, 32, 347-356.

OMB Clearance Supporting Statement

Evaluation of OWH HIV Prevention Program at Minority Institutions Page 16