Revised Supporting Statement Part B (6-2012)

SUPPORTING STATEMENT FOR
PAPERWORK REDUCTION ACT SUBMISSION

PART B

Bureau of Educational and Cultural Affairs

Office of Policy and Evaluation

Evaluation Division:

Gender Assessment Surveys

OMB Control Number: 1405-XXXX

SV2011-0027, SV2011-0028, SV2011-0029, sv2011-0030

2. Collections of Information Employing Statistical Methods

1. There is no sampling for this information collection, as the potential respondent universe for this information collection will be all 778 exchange participants from the ILEP, Fortune, IRG, or Fellows programs for the years between 2003 and 2010. The anticipated response rate for this collection is 60-65%.

2. This information collection will consist of four similarly structured surveys that will be administered to participants from one of the following programs: FORTUNE, IRG, American Fellows, or ILEP that were conducted between the years of 2003 and 2010. The statistical methodology used will be via census. This information collection will only be conducted one time as part of this effort.

3. All ECA/P/V data collection methods are tailored to fit the prevailing political, cultural, safety, security, and accessibility conditions in each country in which participants are located. Successfully contacting and achieving the highest possible response rates are the goals of survey administration. Our methods will include:

• Updating Contact Information Prior: Contact information for program participants will be updated through online searches. All research methods in this effort will be as non-invasive as possible and only information which is publicly available on the Internet will be used. Personal websites, blogs, public message boards such as Yahoo Groups, and 3rd party directories will be used, although we recognize that some of them might contain obsolete information. These sites usually list the time of their most recent update. If the update was over 2 years ago, we will disregard the information or look for additional evidence of its relevance, such as a current news article listing the same person and organization.

• Customized Intro: Customized lead emails will be sent at the start of survey administration to encourage respondent cooperation. The email will inform them about the evaluation, and explain that the survey is voluntary. The email will also provide ways for respondents to contact either the evaluation’s contractor or the Evaluation’s Project Manager here in ECA/P/V with any concerns or questions about the evaluation.

• Informing the Grantee Organizations Prior: Many program participants continue to be in communication with the grantee organization that administered their exchange program long after the program has ended. Informing the grantee organizations in advanced will allow the grantees to vouch for the survey requests that get sent out by the contractor. Doing this will only serve this purpose in the event any of the participants contact the grantee regarding any doubt as to the legitimacy of the initial introductory email that will be sent by RSI. No other information about the participants themselves will be provided to the grantee. A general email communication will be sent out by the grantees encouraging program participants to respond to the survey questionnaire if necessary.

• Non-English Approach: Having multiple langauges will allow respondents who do not have the English capability to both understand and participate in this evaluation. We will translate the American Fellows survey into Spanish and Brazilian Portuguese in addition to offering it in English. Because the IRG program does not have an English speaking requirement the IRG survey in addition to being offered in English will also be translated into no more than 10 foreign languages. These languages are Dari, Arabic, Brazilian Portuguese, Khmer, Spanish, Russian, Bahasa, Malay, Serbian, and Turkish. These languages were selected because a large number of the program’s participants speak these languages and offering this set of 10 foreign languages beyond just English potentially ensures adequate representation across the various regions of the World (as defined by the State Department’s regional structure). Surveys will be translated and back-translated to ensure the meaning of the questions does not vary across the multiple languages offered. Open-ended responses to these two surveys likewise will be translated back into English.

The ILEP and Fortune program participants have strong English skills as the respective programs require and thus these two surveys will only be conducted in English.

• Multiple Attempts and Reminders: A limited number of follow-up reminders will be sent periodically to non-respondents to encourage them to respond.

• Pre-testing Survey: Pre-testing the surveys is extremely useful for clarifying instructions and questions, refining the response categories, as well as ensuring clarity, brevity, relevance, user-friendliness, understandability, and sensitivity to a respondent’s culture and the political climate in which they live. This in turn will allow minimum burden to respondents and encourage them to complete their survey. The Fortune, ILEP and Fellows surveys were pre-tested prior to this submission using between 2 and 5 current and/or past participants.

Using such methods as listed above has in our previous experiences increased response rates.

Additionally, the data collected is only representative of the evaluation’s respondents and therefore all analysis of results and future reports will be clearly linked to only the universe that was surveyed.

4. In order to help develop and refine survey instruments, formative interviews and cognitive interviews were conducted with past program participants for three of the program surveys. Each survey had 5 or fewer formative and/or cognitive interviews conducted.

One-on-one formative, phone interviews were conducted in order to better understand participants' experiences with the program and garner potential ways in which they could implement the newly acquired knowledge and skills upon return home. The point of this series of interviews was to assist with development of survey questions and appropriate response categories.

Cognitive interviews were one-on-one interviews where participants were probed about their interpretations and answers generated in response to the questionnaire. The point of these voluntary and confidential interviews was to get enough information about the respondents’ comprehension and preparation of a response to assess whether they understood the questions and responded as the researchers intended. A secondary goal of this exercise was to garner information to improve the construction and administration of the questions. This research activity included the participants’ evaluation, review, and completion of the survey questionnaire prior to the interview, followed by participation in a telephone or Skype interview for the “think-aloud” exercise. Individual questions were tested for clarity, relevance and comprehensibility. Typical probes in this research included questions such as: How easy or difficult was it to respond to this survey? Did you have difficulty with the meaning of any of the questions? What, if anything, was confusing? Did these response categories make sense to you? What, if anything, was missing from this list? Did you have any difficulties answering this question? The information and feedback from the respondents were used to refine the survey instrument, identify additional response categories, and pinpoint potential changes needed to make the instrument more precise in its language and meaning for the broader group of target respondents.

5. The ECA/P/V individual managing this evaluation’s external contractor (RSI) who will be collecting the data and analyzing the information is Melinda Pitts, 202-632-6314.

3