B. Collection of Information Employing Statistical Methods.
B.1 Respondent Universe and Sampling Methods
There are six respondent types participating in this evaluation project. They are program directors, program staff, program partners, program participants, parents of program participants and community residents. No sampling methods are used for any respondent type. Instead a convenience sample will be used since data is being collected from all grantee staff and program participants. Moreover, program directors, program staff, and program partners of funded organizations are predetermined based on awards made in 2009 by DHHS/OWH. Therefore, the entire universe of respondents for these groups consists of the program personnel and partners of organizations that received grants for this project.
The respondent universe for program participants consists of those girls and female adolescents between the ages of nine and 17 years who are recruited for and participate in the organization’s Girls at Greater Risk program. It is expected that annually 750 girls and adolescent female participants will be recruited among the 10 grantee organizations. It is expected that 80 to 100 percent of program participants will participate in the pre and post-test data collection. This estimate is provided given the requirement of programs funded by the OWH Girls at Greater Risk program to retain 80 percent of their program participants. Grantees will use a variety of strategies to maintain program participation, including individual one to one meetings between participants and staff; maintaining contact with the participants’ parents or guardians; and holding community events and activities.
B.2 Procedures for the Collection of Information
There are several data collection activities included in this project. Appendix G provides a description of the information collected, rationale and sources/references for each data collection used in this project. The following presents data collection procedures for the evaluation project:
Verify IRB approval from all institutions. GEARS will collect IRB approval letters from all institutions participating in the evaluation.
Conduct site visits to all grantees and provide technical assistance on how to obtain consent/assent, administer questionnaires and how to store data.
Obtain OMB clearance.
Finalize all forms with the OMB clearance number printed on forms.
The day OMB clearance is received, GEARS will send an email to each program to inform them that: 1) OMB clearance has been obtained and 2) that they will be receiving within the next three days via email data collection forms and a database with instructions on how to input data. They will also be informed of the dates on which a webinar will be given on how to use the database.
Two weeks after receiving OMB clearance, grantees will participate in a webinar conducted on how to enter data into the data repository. This webinar will be repeated once a month for two months. In addition, ongoing technical assistance will be provided for grantee staff who are entering data into the database.
Two weeks after receiving OMB clearance, GEARS will begin training sessions for GEARS staff on details of the administration of the process evaluation interviews. This two-hour training will review the purpose of the process evaluation and how the interview fits within the evaluation framework. Each process interview question will be discussed and questions answered. This training will be led by GEARS’ evaluation team Senior Evaluator. GEARS staff is trained in interviewing administration and have conducted interviews as part of GEARS’ research and organizational development activities.
One month after OMB clearance begin conducting monthly conference call in order to monitor and answer questions about data collection procedures.
Approximately one to two months after OMB clearance, GEARS will attend the OWH Grantee Meeting and will provide technical assistance to grantee staff on data collection from participants and on data entry. Emphasis will be placed on the consenting process that has been approved by their IRB’s and utilizing the script developed for creating the unique identifier. (Appendix E)
On a quarterly basis throughout the project period, respondents (program staff) will enter the data into Excel spreadsheet and transmit results to GEARS. Program staff will also use submitted data as part of their local evaluations.
Submission of Data
Data collected for this evaluation are designed to fit within existing program data collection activities as required by the grant. Management Information Systems (MIS) information as well as data markers required for the evaluation will be entered into a user friendly Internet based, data repository on a quarterly basis. GEARS Senior Evaluator will make a site visit of all programs and conduct focus groups with a subsample of program participants and with a subsample of parents. During these site visits the GEARS Senior Evaluator will also provide technical assistance regarding data collection and submission.
Process evaluation data will be collected twice a year. These data will be collected by telephone interview and during site visits made for data collection purposes (i.e. conducting focus groups). GEARS staff will ask the respondent permission to audiotape the process evaluation interview. Audio-taping is requested in order to ensure that the detailed information provided by the respondent in this interview is accurately captured in the evaluation data. Specifically, GEARS staff will check the accuracy of the recorded interview responses against the audio-taped interview. All audio-tapes will be kept in a locked filed in GEARS offices and will be destroyed at the conclusion of the evaluation project, when all data and data coding have been finalized.
B.3 Methods to Maximize Response Rates and Deal with Nonresponse
The questionnaire administered to program participants 12 months after the post-test is the only data collection that is not being collected as part of program activities. In order to increase response rates for this data collection OWH grantees will utilize best practices for working with youth which require that the program continue to maintain contact with girls and adolescent females when they are not actively participating in the program. These methods include inviting participants to community outreach events, reunion/alumni parties and holiday events; providing incentives; and keeping in touch through program newsletters, birthday cards, telephone calls (cell phones, texting reminders); and the program’s Facebook page (or other social networking tools).
One hundred percent participation is expected from OWH grantees in collecting process evaluation data. This data will be collected during telephone interviews and as part of the data collection site visit.
B.4 Test of Procedures or Methods to be Undertaken
The majority of the Instruments and items chosen for this evaluation have been selected from standardized instruments previously used or recommended by federal organizations, previously OMB approved evaluations, research groups, or academic institutions with either community-based populations or organizations and tested for their cultural and linguistic appropriateness.
Additionally, two sets of respondents (program directors and program staff) had opportunities to input on development of the instruments and submit copies of instruments they currently use in their local evaluations to guide development of instruments used in this project. Every effort was made to ensure that data collected for OWH was also data needed by grantees for their local evaluation.
B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
Program Development Contact
Aleisha Langhorne, MPH, MHSA
Health Scientist
U.S. Department of Health and Human Services
Office on Women's Health
202-401-8325
Data Collection/Analysis and Statistical Contact
Deborah Brome, Ph.D.
Vice President and Director, Evaluation & Applied Research
Global Evaluation & Applied Research Solutions (GEARS Inc.)
617-328-5141
Michael Milburn, Ph.D.
Professor of Psychology
University of Massachusetts, Boston
100 Morrissey Boulevard
Boston, Massachusetts
617-287-6386
Michael.Milburn@getingears.com
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Funn, Sherrette (OS/ASA/OCIO/OEA) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-28 |