YTYA Supporting Statement (5-2013)

SUPPORTING STATEMENT FOR
PAPERWORK REDUCTION ACT SUBMISSION

Bureau of Educational and Cultural Affairs

Office of Policy and Evaluation

Evaluation Division:

Young Turkey/ Young America Evaluation (YTYA) Survey

OMB Control Number: 1405-XXXX

SV2013-0001

1. Justification

1. The Department of State (DOS) Bureau of Educational and Cultural Affairs (ECA), Office of Policy and Evaluation, Evaluation Division (ECA/P/V) is requesting a new information collection to conduct a survey in support of an upcoming ECA evaluation of the Young Turkey/Young America (YTYA) Program.

This new evaluation will assess the effectiveness of YTYA in achieving its stated goals and objectives, and assess the outcomes of this two-way, bilateral exchange program that has included almost 235 young Turkish and young American professionals between 2009 and 2011. The program was first announced by former Secretary Clinton in 2009 to provide opportunities for emerging Turkish and American leaders (ages 24 to 35) to advance critical dialogue and understanding. Its purpose is to develop participants’ leadership skills, and facilitate ongoing professional collaboration, through joint projects, creation of networks and other initiatives. Between FY 2009 and FY 2011, YTYA offered grants to U.S. non-governmental organizations to design and manage the YTYA exchange program, under one of the following program themes:

• Project Theme One: Foreign Policy Dialogue Among Emerging Leaders: These projects link young Turkish and American leaders in substantive foreign policy dialogue, on issues of importance to both countries.

• Project Theme Two: Social and Economic Challenges for Future Leaders. These projects will expand the capacity of nascent grassroots or not-for-profit organizations working in or with disadvantaged communities to address existing socio-economic issues and challenges.

This survey will review the experiences of Turkish and American professionals who participated in the program under grants funded between FY 2009 and FY 2011. It will assess changes in knowledge and attitudes; probe ways in which they shared what they learned within their communities and the organizations where they work; assess whether and how social networking contributed to the effectiveness of their communications; and examine whether the international exchange experience factored into their subsequent educational and professional choices.

The data captured will help the Department and ECA Bureau successfully meet organizational performance and accountability goals established through the mandates contained in the following authorities:

• Mutual Educational and Cultural Exchange Act of 1961, as amended (also known as the Fulbright-Hays Act) (22 U.S.C. 2451 et seq.)

http://uscode.house.gov/download/pls/22C33.txt

• Government Performance and Results Act of 1993 (GPRA)

http://www.whitehouse.gov/omb/mgmt-gpra/gplaw2m.html

• Government Performance and Results Modernization Act of 2010

http://www.whitehouse.gov/omb/performance/gprm-act

• OMB Memo M-10-01, Increased Emphasis on Program Evaluations http://www.whitehouse.gov/sites/default/files/omb/assets/memoranda_2010/m10-01.pdf

As stated in the memo, “OMB will work with agencies to make information readily available online about all Federal evaluations focused on program impacts that are planned or already underway” as part of a three-pronged effort to strengthen government-wide program evaluation efforts. The guidance noted that public availability of program evaluation information will promote transparency, since agency program evaluations will be made public regardless of the results.

2. The primary purpose of this information collection is to provide ECA/P/V with the ability to assess the Young Turkey/Young America Program in accordance with GPRA, as well as OMB Guidance and Executive Orders. The assessment will provide ECA with data it currently does not have, and will inform ECA senior management and program managers on the design, implementation, monitoring, follow-up and outreach related to this and similar bilateral exchange programs. Thus, findings will be used to improve existing and inform ongoing and future exchange programs in ECA.

This study will assess achievement of program goals only to the extent to which they are reflected in the major research questions outlined in Tables 1 and 2 below. The table below lists the major research questions developed for this evaluation, the outcome measures that may be assessed, and provides contextual information for understanding the affects of this exchange program. The data source that will be used to answer all of the major research questions will be via the on-line survey questionnaire.

Table 1: Research Question Focus Areas

• Exchange Activities

• Participant Knowledge and Understanding

• Participants’ Professional and Leadership Skills

• Participants’ Professional and Career Path

• Participants’ Interest/Involvement in Policy Issues

• Institutional, organizational- and/or community-level outcomes

• Linkages and Communication

• Sharing Program Experiences

• Participant Demographics

Table 2: Research Questions and Outcome Measures
Major Research Questions	Outcome Measure
Exchange Activities: In which types of activities did YTYA young professionals participate during their exchange, and through projects and collaborative activities? Which activities did participants state contributed most significantly to their overall YTYA experience?	Answers to questions regarding focus of participant exchanges and range of projects carried out, and level to which they contributed to the YTYA experience.
Participants’ Knowledge and Understanding: To what extent and in what ways did participation in the YTYA program influence participants’ knowledge, understanding and perception of the U.S. and Turkey, the bi-lateral relationship between the two countries, and development issues/policies in their countries?	Answers to questions regarding changes in participants’ knowledge and understanding of issues relevant to the U.S.-Turkey bilateral relationship, including cross-cultural understanding, foreign policy, and awareness of key development issues and policy areas.
Participants’ Professional and Leadership Skills: How and to what extent did participation in the YTYA improve the professional or leadership skills of participants, particularly in areas such as public discourse, cross-cultural dialogue and collaboration, NGO strengthening and/or web-based communications?	Answers to questions regarding changes in professional and leadership skills, including understanding and mastery of on-line technologies and access to expanded sources of information.
Participants’ Professional or Career Path: To what extent, if any, did YTYA influence participants’ professional or career paths?	Answers to questions regarding whether participation in the YTYA program may have led to changes in participants’ educational and career plans, including decisions to pursue professions related to the U.S.-Turkey bilateral relationship or broader Middle East issues.
Participants’ Interest/Involvement in Policy Issues: To what extent, if any, did YTYA influence participants’ personal interest, involvement or leadership role in any policy, cross-cultural, advocacy or NGO development issues?	Answers to questions regarding changes in participants’ interest and involvement in policy issues, such as advocacy, volunteerism or work in the non-profit sector, both domestically and on an international level.
Institutional, Organizational- and/or Community-level Outcomes: What have been the institutional, organizational- and/or community-level outcomes of YTYA through both the exchange and YTYA projects? What types of issues have been addressed and for whose benefit? Were there end products and to what extent have projects been sustained?	Answers to questions regarding the identification of projects carried out under YTYA, the impact they had and whether they have been sustained. Also an identification of any outcomes, such as strengthening or creation of NGOs, enhanced public awareness of issues, or the creation of new initiatives or products, such as websites, written materials, or conferences.
Linkages and Communication: How have participants communicated with each other and which platforms or tools have they used? What has been the focus of their communications? Since completing the program have they continued communicating with other YTYA participants or contacts they made through the program?	Answers to questions regarding the focus, frequency and purpose of communication with other YTYA participants and program contacts, and the effectiveness of social media in supporting this communication.
Sharing Program Experiences: How have participants shared their program experiences with others in their communities, including peers, colleagues and friends? Have they shared their experiences more widely, such as with the general public, the press or via publications?	Answers to questions regarding ways YTYA experiences were shared with others, including colleagues, friends and the general public.
Participants’ Demographics: Who were the participants in the YTYA program and what types of organizations or constituencies did they represent? What were their motivations for participating?	Answers to questions regarding participants’ institutional affiliation and/or employment and reasons for participating.

Analysis of all collected data will include descriptive statistics and frequencies, providing survey results related to participant background characteristics and project activities, frequency and means of participant communications, knowledge and skills acquired, creation of new partnerships and cross-cultural linkages, and sharing of YTYA experiences in home communities and workplaces. Cross-tabular analysis of survey responses will be conducted to assess any significant variation across different types of participants or differences in program characteristics/experience. Examples may include comparing those participants in Project Theme 1 (Foreign Policy Dialogue among Emerging Leaders) vs. Project Theme 2 (Social and Economic Challenges for Future Leaders). We also anticipate analyzing differences in particular participant-level effects (e.g., satisfaction, continued engagement with colleagues, commitment to support goals of program) across different cohorts to determine how these effects may vary based on length of time elapsed since program participation/completion.

We will not report any finding when n is less than or equal to 5 in order to protect respondent confidentiality and to ensure we are not reporting invalid results.

3. The information collection surveys will be entirely web-based to ease any burden on the participant. The survey will be distributed using the survey application Vovici. Participants will be informed of the survey via e-mails and e-mail reminders that will provide instructions for how to access the survey electronically.

4. Currently, no duplicative information exists, and there is no other reliable method for ECA to collect the information needed to fulfill the GPRA mandates.

5. Information collected under this collection will have no impact on small businesses and other small entities.

6. If the information is not collected, ECA will be unable to complete this study, or gather data requested by ECA senior leadership in order to assess and report on these types of bilateral exchange programs, which are part of the new strategic plan of ECA. Moreover, the Department will be unable to comply fully with its congressional and Department executive mandates, including the GPRA Modernization Act of 2010, which requires the Department to evaluate and report the results of its exchange programs.

7. There are no special circumstances.

8. ECA/P/V has solicited public comments on this collection via a 60-day Notice published in the Federal Register on February 8, 2013 (78 FR 9448-9449). One comment was received. Upon reviewing the comment, ECA/P/V determined that the comment was unrelated to the information collection, and instead addressed broader Department wide policy and budget regarding the program. ECA/P/V has consulted with an external contractor, EurekaFacts about the surveys design, methodology, analysis, and data collection approach.

9. No gifts or payments will be made to the respondents.

10. No promises of confidentiality will be made to respondents.

11. No questions of a sensitive nature are asked in the survey.

12. It is estimated that the total annual hour burden will be 77 hours for the 235 respondents that make up the census population (as explained in Section B1, it is estimated that the response rate to the surveys will be 65%). The total burden was calculated with the expectation that 65% of respondents will complete each survey at an average of 30 minutes each (77 total hours). Because this survey will only be conducted once, the three year total is the same as the annual total.

This survey was pre-tested prior to this submission using four (4) YTYA participants. Burden hours took this into account, as well as the total number of questions and the number of open-ended questions, as well as experience on previously conducted evaluations.

Table 3

Respondent Burden

ITEM	ANNUAL TOTAL	3 YEAR TOTAL
Estimated Number of Respondents	235	235
Estimated Number of Responses	153	153
Average Hours Per Response	30 Minutes	30 Minutes
Estimated Hours for Responses	77	77

To determine the estimated income per hour, the Bureau of Labor Statistics (BLS), “Table 1 Summary: mean hourly earnings and weekly hours for selected worker and establishment characteristics” were reviewed (http://www.bls.gov/ncs/ncswage2010.htm#Overview).  The specific data table is located at http://www.bls.gov/ncs/ocs/sp/nctb1475.pdf.  Due to the unavailability of similar wage rates for Turkey, the U.S. earnings are used below as reference for both U.S. and foreign participants.

Because the participants of this program are young professionals most of whom work in government or NGOs, the earnings referenced for this calculation from the aforementioned table are: average mean hourly civilian earnings are $21.29; private industry workers are $20.47, state and local government workers are $26.08.  Averaging the three totals is $22.61 (weighted by a factor of 1.4 to $31.65, rounded to $32). 

Table 4

Annualized Cost to Respondent for Hours Burden

Description of the Collection Activity	Estimated Total Annual Burden on Respondents (Hours)	Estimated Average Income per Hour	Estimated Hour Burden Cost to Respondents
Web Survey	77 hours	$32	$2,464

13. There are no costs incurred by respondents.

14. The estimated annualized cost to the Federal Government for this collection is $100,181. This number was calculated based on the contractor’s labor for associated tasks, as well as salary of ECA/P/V staff who manage the contractor (as broken down below):

• The data collection budget for this evaluation survey is approximately $60,213. This includes contractor labor for 5 persons for drafting and finalizing the survey instrument, survey programming in the surveying system, survey administration including sending out survey reminders, and producing regular response rate reports, participation in status meetings with ECA/P/V, as well as fees for the software/server expenditures. 

• The contractor’s analysis and reporting budget the data collected through this collection is approximately $31,606 and will include contractor labor for 4 people to do the analysis, report writing and materials, and briefings.

• The approximate cost for ECA/P/V employee time is $8,362. This is calculated as 2 employees (GS-9 and GS-13), with a loaded (or weighted by 1.4) average hourly wage of $47.18, who will spend approximately 5 percent (during about 10 months) of their time (which will equate to about 84 hours each) in providing oversight and additional guidance including reviews of survey, data administration, data analysis, and the report, as well as participation in status meetings with the contractor. Two employees (GS-12 and GS-13) will spend approximately 80 hours preparing documents, obtaining clearances, making revisions for information collection request purposes with a loaded (or weighted by 1.4) average hourly wage of $54.98.

15. This is a new collection.

16. Survey data collection is estimated to begin immediately after OMB approval is received. It is estimated the data collection period will take at least 6 weeks. Following the data collection period, the external contracting firm (EurekaFacts) will conduct basic descriptive analysis (such as frequencies) and cross-tabular analysis as needed (as explained per section A2). The contractor will develop a report for review and approval by ECA.

Once approved by the ECA Assistant Secretary, the evaluation report will be posted on the Department of State, ECA Evaluation Division public site at http://exchanges.state.gov/programevaluations/completed.html. Additionally, an appropriate distribution list, which will include key stakeholders and other organizations and individuals that may be interested in the evaluation results, will also be developed. They will receive notification of the release of this report via email. The contracted evaluators are also required to present results of the evaluation to key stakeholder groups as requested by ECA for a period of time following the evaluation’s completion. Results for this evaluation are estimated to conclude about 9 months after the data collection period has ended.

17. ECA/P/V will display the OMB expiration date.

18. There are no exceptions requested for this collection.

2. Collections of Information Employing Statistical Methods

1. There is no sampling for this information collection, as the potential respondent universe for this information collection is all 235 program participants from the Young Turkey/Young America Program. YTYA participants are both Turkish citizens and American citizens who were involved in the program while in their home country and in their host country, and funded under FY 2009 to FY 2011 grants. The participants from this program have never been surveyed by State/ECA. The anticipated response rate for this entire collection is 65%. This number is based on experience with previous State Department studies that have been completed and the fact that most participated in the program fairly recently.  

2. This information collection will consist of one electronic survey. Given the low total N in the program we’re surveying, we believe that sampling would likely yield an insufficient number of responses. Therefore we are surveying the entire population rather than a specified sample. This information collection will only be conducted one time as part of the Young Turkey/Young America Evaluation.

3. All ECA/P/V data collection methods are tailored to fit the prevailing political, cultural, safety, security, and accessibility conditions in the United States and in Turkey. Successfully contacting and achieving the highest possible response rates are the goals of survey administration. Our methods will include:

• Customized Notification Emails and Letters: Pre-notification emails and/or letters will be sent to all program participants approximately 3 weeks prior to the survey launch to encourage respondent cooperation. These emails/letters will inform program participants of the upcoming online survey, and contain endorsements and mentions of State Department or grantees. This email will inform them about the evaluation and will also provide ways for respondents to contact the firm managing the evaluation on behalf of State/ECA, should they have any concerns or questions about the evaluation, or should they wish to provide an alternative email address to receive the online survey.  As needed, the contractor will send follow-up emails to individuals to resolve any questions or discrepancies in participants’ names, email addresses, or dates of program participation. A second customized introductory/invitation email containing a link to the online survey will be sent at the start of survey administration.

• Participant Contact Information Verification: Extensive contact lists for the program were requested from the respective administering grantee organizations and State Department program office to establish baseline participation in the program and to obtain an initial set of contact data. In addition, ECA/P/V queried available State Department self-registered (i.e., provided by participants) alumni databases to obtain any additional or updated contact information in order to ensure that the contact lists are as accurate as possible.

• Informing the Grantee Organizations: Many program participants continue to be in communication with the grantee organization that administered their YTYA exchange program. Grantees have been notified of the upcoming survey, in the event that the grantees are contacted by former participants from the program.

• Survey Reminders: Besides the initial introductory/invitation email at survey launch, up to three follow-up reminders will be sent to non-respondents to encourage them to respond over the course of the administration period, including a final reminder as the survey comes to a close that will indicate the urgency. Response rates and survey user feedback will be closely monitored and recorded throughout the entire survey administration period to ensure a satisfactory response. ECA/P/V will also be ready to make a judgment call based on response rate status throughout the administration period to both extend the administration period as deemed fit, as well as send an additional final reminder.

• Pre-testing Survey: Pre-testing the survey with four (4) participants was extremely useful for clarifying instructions and questions, refining the response categories, as well as ensuring clarity, brevity, relevance, user-friendliness, understandability, and sensitivity to a respondent’s culture and the political climate in which they live. This in turn allowed the survey’s questions to be designed and refined in a way that minimizes the burden to respondents and encourages them to complete their survey.

Using the data collection methods described above has in our previous evaluations ensured clarity about and transparency in the survey process and helped stimulate higher response rates.

This data collected is only representative of the evaluation’s respondents and all analysis of results and future reports will be clearly linked to only the universe that was surveyed. We will monitor the potential for non-response bias, including tracking response rates by cohort over the collection period and reviewing both respondent and non-respondent demographics. These factors will be taken into account in our analysis and reporting of results, especially when disaggregating the data according to key demographics for which the number of respondents may be less than ideal.

4. To enhance the questionnaire’s design, during the survey pre-development phase the contractor conducted formative interviews with eight (8) former program participants. The formative interviews helped EurekaFacts to better understand program participants’ experiences, including the full range of activities, interactions, roles, and outcomes associated with program participation. These formative interviews allowed the contractor to develop the draft survey.

After the initial draft was developed four (4) of the past program participants completed a test version of the on-line survey. Three of these pre-test participants indicated no issues, while the one participant who suggested minor changes participated in a follow-up interview. Based on this interview and a review of data from the four surveys to ensure wording was clear, conveyed its intended meaning, contained realistic and mutually exclusive response options, and presented scaling of magnitude, agreement/disagreement, etc., that is relevant and understandable to the respondents, minor revisions were made to the survey instrument.

5. The ECA/P/V individual who will be able to answer questions regarding this evaluation is Robin Silver.

8