SS Part B

IAATP-Pre-Test

Supporting Statement – Part B

1. Respondent universe and sampling methods

The target population consists of all trainees who received IAATP training.

As in the first phase of the cross-site evaluation (FY 2006-2011), the sampling design for this final phase of the study (FY 2012-2014) is one-stage cluster sampling (Levy & Lemeshow, 1999). The primary sampling unit (PSU) is the training sessions without stratification. These PSUs were constructed as the sampling frame with information collected from IAATP grantees. A simple random sampling was conducted based on the sampling frame on training sessions as the PSU. No further sampling is done within training session. One-stage cluster sampling is preferable in terms of cost and efficiency, because all registrants in a sampled training session are required to respond to the survey, rather than data collection from individual registrants across sessions.

One-stage cluster sampling does not assume the same number of trainees across sessions. This is an advantage of the sampling design to deal with the problems of attrition and non-respondents. The study is a pretest-follow-up comparison and attrition is inevitable in the study. The final analysis is based on respondents who have data on both pre and follow-up tests. In the one-stage cluster sampling, since the PSU is the training session, the problem of attrition and non-respondents within sessions will not be as serious as that in a random sampling on individuals. The attrition and non-respondents within sessions can be excluded from analysis, assuming no difference in demographic factors between the respondents and non-respondents.

Because the focus of this phase of the study was on mean differences in pre-post training gains among training formats and curriculum, the sample size estimation was based on these differences. The null hypothesis was that the mean of the gain score (the difference between pre and post scores) was equal across treatment conditions (i.e. across curricula or across training formats).

Optimal Design Software was used to conduct the sample size calculations. Calculations assumed a cluster-randomized design because all treatment differences (i.e. training format, curriculum) were applied to the training itself.

Individuals included in the national cross-site evaluation class sample are identified to local grantees and evaluators to ensure that they will not be included in local data collections that would duplicate or fall within the timeframe of the cross-site data collection. Identifying and eliminating duplication in this prospective manner has fostered cooperation among all stakeholders (i.e., grantee staff, trainers, and evaluators) and fostered greater response rates among the trainees selected to complete the cross-site instrument.

2. Procedures for the collection of information.

Sample size estimation in two-level cluster randomized trials is based on required confidence interval, between cluster variance of the outcome variable, the effect size between treatments, the average number of participants per cluster, and the specified power of the study.

Rather than use standard effect size scores (Cohen, 1988), likely effect sizes for differences in gain scores across curricula were calculated from the first phase of data collection based on gains in knowledge, attitude, and behavior. Cohen’s d values among these outcome variables ranged from Differences in gain scores in this data set were .6, with a standard deviation of 7.5, resulting in a value of d = .08 to d=.29. As minimally clinically significant differences should be somewhat larger, d values of .3, considered small effect sizes (Cohen, 1988) were used as conservative estimates.

Assuming a conservative between-cluster correlation= 0.1 (actual between-cluster variance ranged from 0 to .3 across outcome variables, with all but one outcome showing a variance of 0), a 95% confidence interval (z = 1.96), an average of 15 participants per training session, and the required power of 80% (1- = 0.8), the required sample size is 58.

Specifically, the number of sampled clusters is estimated in Figure 1:

where:

 is the probability of a Type I error;

n is the number of participants per training session (cluster);

 is the effect size;

is the between cluster interclass correlation.

Therefore, approximately 58 training sessions per year are needed for the study. Because the change in effect size may dramatically influence the sample size, three different values are shown.

3. Methods to maximize response rates and deal with nonresponse

Pretest Survey techniques

All IAATP trainees complete a pretest as a prerequisite to beginning the training, regardless of whether they are included in the national cross-site data collection. For national evaluation purposes, the pretest will be administered to all trainees registered in 58 sessions selected for the cross-site sample. Based on our anticipated response rate of 90%, and average class size of 15, a total of 870 respondents will be sampled each year. (Note: As explained in Item 4 of Section A, “Efforts to Identify and Eliminate Duplication,” the cross-site pretest is available to the IAATP grantees to use in local evaluation efforts). The baseline survey is administered in paper form onsite prior to the start of training.  This continues to mirror the grantees’ approach, in which a pretest is administered prior to the start of each training class.  The completed forms are copied and then shipped to the national evaluation center for entry.  

Follow-Up Survey techniques

Maximizing response rates is most critical during the follow-up administration of the IAATP survey. The methodology is designed to ensure that completion of the follow-up survey is convenient for trainees. The survey process also includes multiple modes of contacting non-respondents to increase the number who will complete the survey.

By administering the follow-up survey soon after training (optimally 90 days post-training), we maximize survey response rates by avoiding high levels of staff turnover, flagging interest or recall of the IAATP training, etc. Administration of the follow-up survey is primarily electronic.  The national evaluator emails a notification to all trainees within sampled classes who completed a pretest, inviting them to complete the follow-up survey instrument using one of two options:  1) by accessing a web-link to an online version of the survey; or 2) accessing an attached survey to complete and return via email, mail or fax to the research team. Electronic testing has allowed respondents the flexibility to complete the survey at the most convenient time with minimal burden and substantially reduces the need for data entry and processing of materials by evaluation staff. 

Follow-up with non-respondents will continue to be conducted by email, phone, and mail as outlined below.

1. Stage One: Email Reminder

Two weeks after the initial survey invitation, a reminder email message is sent to all non-respondents. The reminder message includes a link to the survey website and offers trainees the option to download an attached Word version of the survey to complete and return via email, fax, or mail.

2. Stage Two: Telephone Contact

Two weeks following the email reminder, an evaluation team member contacts non-respondent trainees by phone to determine whether the original and reminder email messages were received. Three attempts are made to reach each trainee by phone. Once reached, the trainee is given the option to complete the survey online through the original web link, to be emailed a Word version of the survey, or to receive a hard copy survey by mail with a postage-paid, return-addressed envelope. A final, alternative mode for conducting the follow-up survey for non-respondents is for research staff to administer the survey via telephone.

3. Stage Three: Follow-up by mail

Trainees that are not reachable by email or phone are mailed the request to complete the survey. The mailing includes a hard copy of the survey and a postage-paid, return-addressed envelope.

Table I provides an overview of the data collection points, format, and timeframes for the pretest and follow-up survey.

Table I

Summary of IAATP Survey Methodology

Data Collection Point	Sample	Instrument	Timeframe	Format
A1	All registrants from a sample of scheduled training sessions	Pretest of adoption knowledge, attitudes, behaviors	Onsite immediately prior to training	Paper form
A2	Same as A1	Follow-up test of adoption knowledge, attitudes, behaviors	90-180 days (3-6 months) after training	Email invitation to complete online survey OR Download attached survey and return completed form via email, fax, or mail (with administration of the survey via telephone and mail for non-respondents)

3. Tests of procedures or methods to be undertaken

The survey methodology described above was initially pilot tested in spring 2008 in order to confirm its viability and to identify possible procedural or methodological challenges in need of attention or improvement. In addition to providing a test of survey item validity, the pilot test was instrumental in confirming ease of administration in terms of the amount of time required to complete the pretest survey, and user access and responsiveness to web-based and alternative methods for completing the post-test survey.

The proposed methodology and specific items in the survey instrument did not require revision as a result of the pilot test. The results of the pilot indicated the survey instruments were able to discern knowledge gains and retention, change in trainee attitudes about adoption, and change in the behaviors and skills applied in the counseling of pregnant women as a result of trainees’ participation in the IAATP training.

Survey instruments were piloted again for the new grant period among respondents who were representative of the population that would be completing the final survey (i.e., individuals in health care settings who provide services to pregnant women and are participating in the IAATP training). The survey items were also revisited with the project directors of the six IAATP grantees, who are both familiar with the training objectives and experienced in testing IAATP trainees. All grantees were provided the instruments to review and participated in a joint meeting to review the surveys question-by-question and state their impressions about the wording of the items.

3. Individuals consulted on statistical aspects and individuals collecting and/or analyzing data

National Cross-Site Evaluation Contractor:

James Bell Associates, 3033 Wilson Boulevard, Suite 650, Arlington, VA 22201; (703) 528-3230.

Statistical Consultants:

Karen Grace-Martin, President, The Analysis Factor, 678 Valley Road, Brooktondale, NY 14817; (877) 272-8096.

Rusan Chen, Senior Statistician, 3520 Prospect Street, Suite 314, Car Barn Building, Georgetown University, Washington DC, NW 20057; (202) 687-6602.

References

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence

Erlbaum Assoc.: New Jersey.

Levy, P. S. & Lemeshow, S. (1999). Sampling of populations (3rd ed.). John Wiley & Sons, Inc: New York.

Appendices

Appendix A: Legislation

Title XII, Subtitle A – Infant Adoption Awareness

TITLE XII—ADOPTION AWARENESS

Subtitle A — Infant Adoption Awareness

SEC. 1201. GRANTS REGARDING INFANT ADOPTION AWARENESS.

Subpart I of part D of title III of the Public Health Service Act, as amended by section 801 of this Act, is amended by adding at the end the following section:

‘‘SEC. 330F. CERTAIN SERVICES FOR PREGNANT WOMEN.

‘‘(a) INFANT ADOPTION AWARENESS.—

‘‘(1) IN GENERAL.—The Secretary shall make grants to national, regional, or local adoption organizations for the purpose of developing and implementing programs to train the designated staff of eligible health centers in providing adoption information and referrals to pregnant women on an equal basis with all other courses of action included in nondirective counseling to pregnant women.

‘‘(2) BEST-PRACTICES GUIDELINES.—

‘‘(A) IN GENERAL.—A condition for the receipt of a grant under paragraph (1) is that the adoption organization involved agree that, in providing training under such paragraph, the organization will follow the guidelines developed under subparagraph (B).

‘‘(B) PROCESS FOR DEVELOPMENT OF GUIDELINES.—

‘‘(i) IN GENERAL.—The Secretary shall establish and supervise a process described in clause (ii) in which the participants are—

‘‘(I) an appropriate number and variety of adoption organizations that, as a group, have expertise in all models of adoption practice and that represent all members of the adoption triad (birth mother, infant, and adoptive parent); and

‘‘(II) affected public health entities.

‘‘(ii) DESCRIPTION OF PROCESS.—The process referred to in clause (i) is a process in which the participants described in such clause collaborate to develop best-practices guidelines on the provision of adoption information and referrals to pregnant women on an equal basis with all other courses of action included in nondirective counseling to pregnant women.

‘‘(iii) DATE CERTAIN FOR DEVELOPMENT.—The Secretary shall ensure that the guidelines described in clause (ii) are developed not later than 180 days after the date of the enactment of the Children’s Health Act of 2000.

‘‘(C) RELATION TO AUTHORITY FOR GRANTS.—The Secretary may not make any grant under paragraph (1) before the date on which the guidelines under subparagraph (B) are developed.

‘‘(3) USE OF GRANT.—

‘‘(A) IN GENERAL.—With respect to a grant under paragraph (1)—

‘‘(i) an adoption organization may expend the grant to carry out the programs directly or through grants to or contracts with other adoption organizations;

‘‘(ii) the purposes for which the adoption organization expends the grant may include the development of a training curriculum, consistent with the guidelines developed under paragraph (2) (B); and

‘‘(iii) a condition for the receipt of the grant is that the adoption organization agree that, in providing training for the designated staff of eligible health centers, such organization will make reasonable efforts to ensure that the individuals who provide the training are individuals who are knowledgeable in all elements of the adoption process and are experienced in providing adoption information and referrals in the geographic areas in which the eligible health centers are located, and that the designated staff receive the training in such areas.

‘‘(B) RULE OF CONSTRUCTION REGARDING TRAINING OF TRAINERS.—With respect to individuals who under a grant under paragraph (1) provide training for the designated staff of eligible health centers (referred to in this subparagraph as ‘trainers’), subparagraph (A) (iii) may not be construed as establishing any limitation regarding the geographic area in which the trainers receive instruction in being such trainers. A trainer may receive such instruction in a different geographic area than the area in which the trainer trains (or will train) the designated staff of eligible health centers.

‘‘(4) ADOPTION ORGANIZATIONS; ELIGIBLE HEALTH CENTERS; OTHER DEFINITIONS.—For purposes of this section:

‘‘(A) The term ‘adoption organization’ means a national, regional, or local organization—

‘‘(i) among whose primary purposes are adoption;

‘‘(ii) that is knowledgeable in all elements of the adoption process and on providing adoption information and referrals to pregnant women; and

‘‘(iii) that is a nonprofit private entity.

‘‘(B) The term ‘designated staff’, with respect to an eligible health center, means staff of the center who provide pregnancy or adoption information and referrals (or will provide such information and referrals after receiving training under a grant under paragraph (1)).

‘‘(C) The term ‘eligible health centers’ means public and nonprofit private entities that provide health services to pregnant women.

‘‘(5) TRAINING FOR CERTAIN ELIGIBLE HEALTH CENTERS.— A condition for the receipt of a grant under paragraph (1) is that the adoption organization involved agree to make reasonable efforts to ensure that the eligible health centers with respect to which training under the grant is provided include—

‘‘(A) eligible health centers that receive grants under section 1001 (relating to voluntary family planning

projects);

‘‘(B) eligible health centers that receive grants under section 330 (relating to community health centers, migrant health centers, and centers regarding homeless individuals and residents of public housing); and

‘‘(C) eligible health centers that receive grants under this Act for the provision of services in schools.

‘‘(6) PARTICIPATION OF CERTAIN ELIGIBLE HEALTH CLINICS. — In the case of eligible health centers that receive grants under section 330 or 1001:

‘‘(A) Within a reasonable period after the Secretary begins making grants under paragraph (1), the Secretary shall provide eligible health centers with complete information about the training available from organizations receiving grants under such paragraph. The Secretary shall make reasonable efforts to encourage eligible health centers to arrange for designated staff to participate in such training. Such efforts shall affirm Federal requirements, if any, that the eligible health center provide nondirective counseling to pregnant women.

‘‘(B) All costs of such centers in obtaining the training shall be reimbursed by the organization that provides the training, using grants under paragraph (1).

‘‘(C) Not later than 1 year after the date of the enactment of the Children’s Health Act of 2000, the Secretary shall submit to the appropriate committees of the Congress a report evaluating the extent to which adoption information and referral, upon request, are provided by eligible health centers. Within a reasonable time after training under this section is initiated, the Secretary shall submit to the appropriate committees of the Congress a report evaluating the extent to which adoption information and referral, upon request, are provided by eligible health centers in order to determine the effectiveness of such training and the extent to which such training complies with subsection (a) (1). In preparing the reports required by this subparagraph, the Secretary shall in no respect interpret the provisions of this section to allow any interference in the provider-patient relationship, any breach of patient confidentiality, or any monitoring or auditing of the counseling process or patient records which breaches patient confidentiality or reveals patient identity. The reports required by this subparagraph shall be conducted by the Secretary acting through the Administrator of the Health Resources and Services Administration and in collaboration with the Director of the Agency for Healthcare Research and Quality.

‘‘(b) APPLICATION FOR GRANT.—The Secretary may make a grant under subsection (a) only if an application for the grant is submitted to the Secretary and the application is in such form, is made in such manner, and contains such agreements, assurances, and information as the Secretary determines to be necessary to carry out this section.

‘‘(c) AUTHORIZATION OF APPROPRIATIONS.—For the purpose of carrying out this section, there are authorized to be appropriated such sums as may be necessary for each of the fiscal years 2001through 2005.’’.

Appendix B:

Best Practice Guidelines for the Infant Adoption Awareness Training Program

Infant Adoption Awareness Training Program

Guidelines

On October 17, 2000 the U.S. Congress, under Public Law 103-310, amended the Public Health Services Act to authorize specific activities pertaining to Infant Adoption Awareness (title XII, Subtitle A). The legislation requires the Secretary of the Department of Health and Human Services (DHHS) to award grants to adoption organizations to develop and implement programs to train the designated staff of eligible health centers in providing adoption information and referral to pregnant women on an equal basis with all other courses of action included in nondirective counseling for pregnant women. The term "eligible health centers" means public and nonprofit private entities that provide health services to pregnant women. The legislation also requires the Secretary to establish a set of best-practice guidelines to which the DHHS-funded training programs will adhere in providing training to staff of eligible health centers.

In response to these legislative requirements, the Department of Health and Human Services has developed the following "best-practice" guidelines to be followed by recipients of cooperative agreements in developing curriculum and providing training to implement the Infant Adoption Awareness Training Program (IAATP). This "best-practice" guideline is critical for attaining the primary IAATP goal- enhancing the ability of health center personnel to provide adoption information and referral on an equal basis with all other courses of action included in nondirective counseling for pregnant women.

Consistent with the intent of the legislation, these guidelines were developed in consultation with 29 experts in the fields of adoption, child welfare, health services, medicine, law, and adoption counseling, as well as adoptive parents.

Guidelines Specific to the Infant Adoption Awareness Training Program

Training Goals

1. The training will impart up-to-date and accurate information about adoption, including the various types of adoptions (e.g., closed adoptions and adoptions involving varying levels of "openness" with respect to the amount of contact or information exchanged between adoptive parents and birth parents).

2. The training will be consistent with applicable State law, imparting information on the legal issues pertaining to adoption, including the rights of the birth mother and father.

3. The training will impart information to the trainees about the Multiethnic Placement Act/Interethnic Placement Act (MEPA/IEP), particularly as it relates to the circumstances under which a birth parent may or may not choose adoptive parents for the child.

4. The training will impart information to the trainees about the Indian Child Welfare Act (ICWA), and particularly will explain that organizations that work with birth parents on adoption will ascertain as soon as possible whether a child is or will be subject to ICWA prior to proceeding with the adoption process.

5. The training will impart information about how family members and the birth mother's community may impact her pregnancy decision process.

6. The training will impart information about the role of the birth father in the pregnancy decision.

7. The training will impart information about various adoption services available within the community and how to assess the quality of those services and their appropriateness for a particular woman.

8. The training will impart information on adolescent development and the differences between counseling adolescents at varying ages and counseling older women.

9. The training will impart information about the psychological and emotional reactions such as shame, grief, loss, guilt, and depression that the birth mother is likely to experience throughout the decision-making process as she considers various pregnancy options, as well as the emotions the birth father is likely to experience.

Basic Skills

1. Trainees will increase their awareness of their attitudes and biases pertaining to adoption so that they are able to present the adoption option in an objective, non-biased manner.

2. Trainees will increase their sensitivity, understanding and skills regarding the influences that both a birth mother and birth father may experience from family, peers, and community.

3. Trainees will improve their basic counseling skills, including cultural competence, listening, building rapport, recognizing someone in crisis, being empathetic and treating clients with respect.

4. Training participants who will counsel pregnant women will be skilled in non-directive counseling to ensure that adoption information, and information about other pregnancy options, is presented objectively, without bias or judgment.

5. Consistent with State and Federal law, trainees will increase their knowledge of adoption and adoption procedures so that they are able to present accurate and up-to-date information during counseling consistent with State and Federal law.

6. Training participants will increase their knowledge of available adoption-related referral resources and how to assess the quality and/or appropriateness of these resources.

7. Trainees who will counsel pregnant women will have basic case management skills, including the ability to assess service needs and make appropriate referrals.

Curriculum

1. The training curriculum will include interactive exercises that promote skills development, such as role-playing and discussions of potential responses to various scenarios.

2. The training curriculum will include exercises, such as attitude awareness activities, that promote awareness of personal biases, prejudices, and negative attitudes and how they impact the provision of adoption information, as well as information on other pregnancy options.

3. The training curriculum will include a component in which birth parents, adoptive parents, and/or adult adoptees present their experiences with adoption.

4. The training curriculum will include resource materials that trainees can take with them to refer to when they are providing counseling on pregnancy options and to disseminate to the women they are counseling.

Training Structure

1. The training should involve no more than 2 (6-hour) days.

2. The training should be conducted by experienced trainers.

3. The training should include presentations and opportunities for interaction with professionals from both the health field and the adoption field.