Supporting Statement SectionB10-16-09v5

Supporting Statement Section B

Table of Contents

October 16, 2009

B. Collection of Information Employing Statistical Methods 25

B. 1. Respondent Universe and Sampling Methods 25

B. 2. Procedures for the Collection of Information 27

B. 3. Methods to Maximize Response Rates and Deal with Non-Response 32

B. 4. Test of Procedures or Methods to Be Undertaken 34

B. 5. Individuals Consulted on Statistical Aspects and Individuals Collecting

and/or Analyzing Data 35

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

This supporting statement addresses two data collection procedures related to the evaluation of the NIAID HIV Vaccine Research Education Initiative (NHVREI): the focus groups and the survey. Only the survey involves the use of statistical methods. Therefore, this section will describe in detail the various aspects of key influencer survey data collection methods.

B.1. Survey Respondent Universe and Sampling Methods

This survey will collect information for an evaluation using a nonequivalent groups design, allowing comparisons between key influencers with different levels of NHVREI contact. Potential key influencer respondents are identified by:

(1) Their association with a NHVREI Partner or Network organization;

(2) Their collaboration with NHVREI Partners; or

(3) Their role in organizations that have had no NHVREI contact but that are matched to Partner, Network, or collaborator organizations.

(1) Partner and Network Staff.

NHVREI staff will provide a list of the 23 Partner and 25 Network organizations. Up to 5 individuals (including the core staff responsible for implementing NVHREI activities) will be sampled from each of the 48 organizations, for a total of 240 individuals. Individuals will fill the following positions (or their equivalents, as applicable):

• Two individuals from board-level positions, including the board of directors, advisory board or the advisory council. Potential respondents will be selected in the order in which they are displayed on the website.

• Two individuals at the administrative/executive level, including executive directors, CEOs, and/or program directors. If the organization is so small that only 1 administrator is involved, an additional board-level member will be selected.

• One individual holding a position as Program Outreach/Health Education/Communications Specialist, or other position that involves direct contact with clients from the community.

(2) Collaborators

Each of the 23 Partners will provide a list of 10 individuals who have received information, training, or have formed partnerships with the Partners, for a total of 230 individuals working at the local level. The level of collaboration between organizations or between a partner organization and an individual may vary. Partners that have focused their efforts on developing partnerships with just a few organizations may select more than one individual from an organization; other Partners focusing on broad outreach to key influencers may provide ten individuals associated with ten different organizations.

(3) No Contact Matches

Each of the individuals working in organizations with a local community focus and known NHVREI contact (18 of 23 Partners, 6 of 25 Network, and all collaborators) will be matched with individuals working in a similar role in an organization with similar characteristics, but with no known or minimal NHVREI contact. The matched organizations are expected to be involved at a local level in HIV/AIDS prevention, advocacy or health education/outreach, serving communities highly impacted by HIV/AIDS in cities without a Local Partner. About 90 individuals from 18 Local Partners will be matched with 90 potential no-contact respondents. Similarly, 30 individuals from 6 Local Network organizations will be matched with 30 persons with no known contact, and 240 collaborators will be matched, for a total of 350 individuals with no known NHVREI contact. Notably, staff from 6 National Partner and 15 National Network organizations will not be matched, since most or all national organizations related to HIV and highly impacted communities have already had contact with NHVREI in some way. See Table B1-1 for a summary of this information.

Table B1-1 Source of Individuals in the Sample

NHVREI Organization Type	Individuals With Some NHVREI Contact	Individuals From Matched Organizations (No Known Contact)	Total Number Individuals
5 National Partners (5 people per org)	25	0	25
18 Local Partners (5 people per org)	90	90	180
Local Collaborator (10 people per Partner)	230	230	460
19 National Network (5 people per org)	95	0	95
6 Local Network (5 people per org)	30	30	60
TOTAL	470	350	820

.

B.2. Procedures for the Collection of Information

NHVREI program staff will request that each Partner and Network organization provide (1) contact information for core staff (i.e., staff funded to perform NHVREI activities), as well as (2) contact information for ten people with whom the Partner plans to collaborate on NHVREI-related activities within the study period. Evaluation staff will identify other potential respondents through a review of the organizations’ Web sites and through the responses of program directors, who will be asked to identify outreach staff within their programs on the survey itself.

Sampled individuals will be sent an e-mail and/or letter requesting their assistance in an online survey related to HIV vaccine research. The recruitment e-mail/letter will indicate that the study is sponsored by the National Institute of Allergy and Infectious Diseases of the National Institutes of Health. The e-mail/letter will succinctly inform the reader of the importance of the survey, as well as procedures for maintaining the privacy of respondents (i.e., the identities of individuals will not be released, identifying information will be stored separately from the survey responses, and all collected information will be analyzed in the aggregate). The e-mail/letter will state that completion of the online survey will result in the release of an incentive. The incentive will be provided immediately on the computer screen as a unique coupon code to be used at Amazon.com. The message will also mention that there will be a follow up survey in about a year. A draft of the e-mail/letter may be found as Attachment E.

The letter/email from NIAID will be accompanied by a letter of support from a national-level organization known to command respect among community-based organizations working with populations highly impacted by HIV/AIDS. The letter will encourage readers to complete the survey, explaining why participation in the survey is important to organizations serving communities highly impacted by AIDS. An example of the letter may be found as Attachment J.

The letter/e-mail will provide the URL for the online survey. The URL will lead to an online survey that has been customized to provide the questions appropriate for the level of NHVREI contact and the organization involved. On the survey instrument attached in Attachment C, items are marked for use with High Contact, Other Contact, and No Known Contact. The instrument has been designed so that for each individual, the survey takes no more than 20 minutes to complete.

Approximately three weeks after the first e-mail or letter has been sent, a second message will be sent to nonresponders reiterating the request for participation. Further follow up via e-mail, postal service, and telephone will be attempted until the 80 percent response rate is achieved.

Approximately 12 months after initial data collection is completed, an e-mail or letter will be sent to individuals who had previously responded to the survey. In instances where an individual has moved from the organization, a survey will be sent to the person in a comparable position at Time 2 if the person has been in that position at least 3 months; otherwise the original respondent will be sent the survey. As with the Time 1 letter, the Time 2 letter will provide an URL for an online survey, along with information about the study, circumstances of privacy, and how to access an incentive.

Power Analysis. There are two key analytic objectives that drive the determination of sample size. The first involves comparisons between groups describing level of contact at Time 1, while the second involves comparisons between changes in scores between Time 1 and Time 2.

Group comparisons. Individuals will be categorized into one of three groups associated with three levels of NHVREI contact:

1. High Contact, involving core staff in NHVREI Partner organizations who have responsibility for implementing NHVREI programs and have direct contact with NHVREI program staff;

2. Other Contact, involving:

• Partner organizations’ peripheral staff and board members,

• Collaborators;

• Network organization staff; and

3. No Known/Minimal Contact, involving staff from local organizations that have no known or minimal direct involvement with NHVREI, but are matched with local organizations known to have had direct NHVREI contact.

Table B.2-1 shows the number of individuals to be recruited for the survey, with the expected yield at Time 1 (with 80 percent response rate) and Time 2 (with a 90 percent followup rate from people who responded at Time 2).

Table B.2 – 1 Survey Sample Numbers and Yield Response Rates

Organization Type	Number Receiving Request to Participate	Number Responding at Time 1	Number With Complete Data at Time 1 and Time 2
		(80% Response Rate)	(90% FU Rate)
High Contact
Core staff from 23 Partner Organizations (2 staff per org)	46	37	33
Total	46	37	33
Other Contact
Peripheral Staff/Board from 23 Partner Organizations (3 staff per org)	69	55	50
Individuals from 25 Network Organizations (5 staff per org)	125	100	90
Collaborators	240	184	166
Total	424	339	305
No Known/Minimal Contact
Individuals from Matched Local Organizations	350	280	252
Total	350	280	252
TOTAL	820	656	590

* Core staff and peripheral staff come from the same Partner Organizations

A key variable for the survey is the proportion of key influencers reporting that they have spoken at least once to friends or family about supporting vaccine research, and there is adequate power to detect differences between the Other Contact and No Known Contact groups at Time 1. Table B.2-2 shows that with the harmonic mean at 306 Time 1 cases in the Other Contact and No Known Contact groups, power to detect a 10 percent difference between proportions at the midpoint is over 79 percent with alpha = .05 one-tailed. Comparisons involving a larger difference, proportions well above or below the midpoint, or those involving a greater number of cases will have greater power.

Table B.2-2 Expected Power to detect the specified difference between No Known Contact and Other Contact (harmonic mean=300) at alpha = .05 one-tailed
Proportion No Known Contact	Proportion Other Contact	H (effect size)	Power to detect difference between P1 and P2 alpha = .05 one-tailed
45%	55%	.2	.79
45%	60%	.3	.98
10%	21%	.3	.98

Comparisons involving the High Contact group will have power to detect only large differences between the groups because of the relatively low number of cases in that group (n=37). Fortunately, a large difference in proportions is expected to exist between the High Contact group and the No Known Contact group at Time 1. As shown in Table B.2-3, if 10 percent or less in the No Known Contact group report a given opinion, and 30 percent or more in the High Contact group report it, the power to detect that difference as statistically significant will be at least 86 percent, well above the standard of 80 percent. If the proportion of No Known Contact cases in the analysis is much higher, power to detect differences is lower. At the midpoint, differences of 24 percent exceed the 80 percent power target by several points, with differences of 20 percent detectable 73 percent of the time.

Table B.2-3 Expected Power to detect the specified difference in proportions between No Known Contact and High Contact (harmonic mean = 65, alpha=.05 one-tailed)
Proportion No Known Contact	Proportion High Contact	h (effect size)	Power to detect difference between proportion alpha = .05 one-tailed
10%	29%	.5	.88
45%	69%	.5	.88
45%	65%	.4	.73

Changes over time. Respondents will be asked to complete a second survey about one year after the first one has been completed. Comparisons will be made between the Other Contact and No Known Contact groups on the number of respondents reporting an increase in positive attitudes or supportive behaviors. If there are at least 305 and 252 cases in each group, respectively, the evaluation will be able to detect at least a 15 percent difference between the two groups at a power of 96 percent or greater at all proportions, as shown in Table B.2-4. Note that if the proportion of cases showing increases in desired knowledge, attitudes, and behaviors in the No Known Contact group is low (about 5 percent), the study will be able to detect a much smaller difference between the groups (between 5 and 13 percent). The 45-55 percent comparison in the last row of the table displays the power (72 percent) associated with the detection of a 10 percent difference between the proportions at n’=250, alpha=.05 one-tailed. This is the minimum power to detect at 10 percent difference for the sample.

Table B.2-4 Expected Power to detect the specified difference over time for Other Contact and No Known Contact (harmonic mean =276, independent sample, alpha = .05 one-tailed)
P1 (No Known Contact)	P2 (Other Contact)	H (effect size)	Power to detect difference between P1 and P2 alpha = .05 one-tailed
5%	13%	.3	.96
10%	21%	.3	.96
20%	33%	.3	.96
30%	44%	.3	.96
40%	55%	.3	.96
45%	60%	.3	.96
45%	55%	.2	.72

B.3. Methods to Maximize Response Rates and Deal with Non-Response

High response rates minimize selection bias in survey findings. Several procedures will be implemented to maximize the response rate. Survey response rates are more robust when the research topic is salient to the respondent’s work, when the questionnaire has been designed for maximum ease of administration, and when the data collection protocol is tailored through a variety of incentives and accommodations to acknowledge respondents’ cooperation and contribution. The presentation of the survey is also important, so that respondents can differentiate it from other mail and research requests.

The introductory letter or e-mail with the link to the survey will indicate that it is sponsored by NIAID, a prestigious NIH institute known by these HIV/AIDS prevention educators to be at the forefront of HIV/AIDS research. The e-mail/letter will succinctly inform the reader of the importance of the survey, as well as procedures for maintaining the privacy of respondents (i.e., identities of individuals will not be released, identifying information will be stored separately from the survey responses, and all information collected will be analyzed in the aggregate). The e-mail will be sent from an NIH address, and letters will be sent on NIH letterhead.

The letter will also discuss the conferment of a monetary incentive or honorarium that sufficiently acknowledges the respondent’s time and cooperation. The incentive is designed to be conferred immediately upon completion of the survey; which is expected to improve completion rates. An incentive of $25 will be conferred. A full discussion of how the incentive amount was determined may be found in Section A.9.

The recruitment letter or email will also enclose or attach a letter from an organization known to command respect among organizations that provide HIV/AIDS services among communities highly impacted by that disease. The letter will encourage participation in the survey by explaining how the information will be used to benefit the community. A copy of the letter may be found in Attachment J.

Online administration of the survey is expected to greatly increase the ease of data collection for these sampled persons, who are expected to be computer literate. Importantly, the incentive for completion of the survey can be released electronically as soon as the survey is completed. The incentive code can be redeemed online immediately at Amazon.com. We expect that gaining immediate access to the incentive rather than waiting for a mailed check will prove attractive to many participants.

The strategy for telephone follow up has been carefully designed and will be staffed with callers trained in refusal conversion. By following up consistently and persistently, NIAID will demonstrate that it is committing time and energy to obtain the most valid data possible by obtaining the opinions of as many persons as possible. The contractor will use refusal avoidance methods during all communications to lessen the need for refusal conversion. For those who do refuse, an experienced refusal conversion interviewer will attempt to collect responses on the questions deemed most critical.

Survey staff will work with the respondent to obtain data in whatever manner is convenient to the participant. Hard copies of the survey will be sent if so required, and the respondent will have the option of calling the contractor to respond to the survey by telephone if necessary.

Consistent with the response rate calculations approved by the American Association for Public Opinion Research (AAPOR), response rates for this study will be calculated as follows:

Number of Completed Surveys

Number of Completed Surveys + Number of Nonrespondents

B.4 Test of Procedures or Methods to Be Undertaken

When constructing the survey instrument, items used previously in other surveys by other NIH Institutes and Centers or organizations were carefully evaluated for inclusion. The survey instrument was tested with cognitive interviews with nine respondents from organizations similar to the ones that will provide respondents for the survey. In response to their comments, questions were revised, dropped or combined, response categories were added to several items, and several small wording changes were made.

A pre-test of sampling procedures was conducted with ten organizations through searches of Web sites and publicly available materials. (No individuals were actually contacted). For the five local and national organizations, we were able to identify board members and program directors, but the names of outreach workers could not reliably be obtained online. Therefore, the survey was revised to enable collection of outreach workers’ names from program directors. Of identified potential respondents, we were able to obtain e-mails for about half of them; addresses for all organizations were available online. Thus, a high proportion of potential respondents will need to be contacted by mail.

A trial of the matching process for five National Partners found that most HIV/AIDS organizations working at the national level have had some contact with NHVREI through a variety of different channels. For this reason, organizations in the matched “No/Minimal Contact” group will limited to organizations focused on local communities.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The contractor collecting and analyzing information for the NHVREI will be NOVA Research Company (NOVA). Responsibility for collecting and analyzing information obtained through the methodologies described above will rest with NOVA. All data collection and analysis will be performed in compliance with OMB, Privacy Act, and Protection of Human Subjects requirements.

24

Evaluation of NIAID’s HIV Vaccine Research Education Initiative