YCC: OMB Supporting Statement, Appendix A
Appendix A
Youth CareerConnect: Recruitment Visit Protocol
Time: 20 minutes
Staff involved: All staff involved for the day (Approximately 5-8)
The purpose of this section is to review the agenda and conduct formal introductions with each of the individuals who will participate in the day’s meetings. The goal is to give visitors and site staff a chance to know one another and understand everyone’s roles in the program and the evaluation.
Time: 60 minutes (20 minutes to present; 40 minutes for discussion)
Staff involved: All staff involved for the day (Approximately 5-8)
The purpose of this section is to provide an overview of the random assignment design for the evaluation and explain the responsibilities that such a study requires of both the evaluation team and program staff. The goal of the presentation is to ensure that key stakeholders at the site understand the need for random assignment and how it works, and to lay the foundation for discussing how to develop random assignment procedures that can fit into their existing recruitment and enrollment processes.
A. STUDY OVERVIEW PRESENTATION
The PowerPoint presentation will cover the key components of the evaluation, which include:
Overview of and need for the evaluation
Summary of the key research questions and what we will learn about the YCC programs
Review of the study design
Impacts versus outcomes
Value of random assignment, its prior use, and why other designs are problematic
Ethics of doing random assignment
How random assignment works
Key principles behind random assignment
Over subscription: ideally we want as many not in program as in it, but a 70 (in program) to 30 ratio might work
Program alternatives that do not overlap
The level of random assignment, which could be at the school, program district, or some other level and which we refer to as “site”
The general process of implementing random assignment
The need for informed consent and good contact information on study participants
What happens to the treatment and comparison groups after random assignment
The importance of monitoring random assignment
Roles and responsibilities of sites and study team
What the sites have to do before, during, and after random assignment
Technical assistance provided before, during, and after random assignment
What Mathematica/SPR must do before, during, and after random assignment
Need for site and Mathematica/SPR to work closely throughout
High-level timetable for study activities
Time for questions
Time: 60 minutes
Staff involved: 2-5 individuals responsible for program oversight, recruitment and enrollment
A. PROGRAM MODEL
Confirm the description of the YCC program provided in the grantee’s application, including the number of participating high schools, districts, program components, and support services in place in 2014 and planned in 2015.
Confirm the program structure for each YCC program, including whether the YCC program is within an existing school or in its own school, the grades covered by the program, and the grades that students can enter the program.
Confirm the employer and higher institution partners who are part of the program and their formalization through MOU’s or LOA.
Confirm the primary factors that distinguish the YCC program from other programs that comparison group students could attend, including the potential for contamination.
B. RECRUITMENT, APPLICATION, AND ENROLLMENT
The purpose of this discussion is to allow the evaluation team to build a flow chart and timeline that depicts the recruitment, application, and enrollment processes and their timing and to identify the number of students in each phase. Discussion will begin with a tailored draft flow chart and timeline developed from the application materials and projected enrollment numbers taken from the performance matrices submitted to DOL.
Confirm the grantee’s recruitment processes, including changes since they submitted their application and any expected changes in 2015.
When is recruitment done and which students and schools are involved in the process?
Is it ongoing or does it occur over a discrete period of time?
What recruitment materials are used, such as brochures and flyers?
Is there variation across your schools or YCC programs?
Confirm the grantee’s application processes, including changes since they submitted their application and any expected changes in 2015.
When do students apply (prior winter or spring or in the fall of the current year)?
Is it ongoing or does it occur over a discrete period of time?
Does the program use or plan to use program application forms?
If so, which staff receive and process these forms?
Is the process centralized or performed separately by school or program?
How many students applied last year and this year? Do you expect this to change?
Confirm the eligibility requirements for the program and any changes expected in 2015.
Is there variation in eligibility requirements across schools or programs?
When is eligibility determination conducted?
Is it ongoing or does it occur over a discrete period of time?
Do the requirements differ by industry/occupational focus?
What percentage of applicants do you anticipate being eligible for the program?
Confirm the grantee’s enrollment processes, including expected changes in 2015.
When do accepted students enroll in the program?
Is it ongoing or does it occur over a discrete period of time?
How are they notified that they are accepted?
What percentage of accepted students do you anticipate enrolling in the program?
Does this vary across schools or programs?
Confirm/discuss the potential for oversubscription for the 2015 school year to accommodate a comparison group for the study.
How oversubscribed is your program (are your programs), for example, 50 percent more applicants than slots?
What processes are used to select students for the program, such as lotteries?
Do you use randomly ordered wait lists for lottery losers?
If so, what is the process for moving students from the wait list into the YCC program?
If not, what process do you use?
Do you expect substantial turnover in your program?
Confirm the grantee’s assessment of program completion for students, and ask about any changes anticipated in 2015. Ensure that they understand that non-completion can be for a host of reasons such as enrolling in a different program, moving out of the program service area, or dropping out of school.
How do you define a program completer (what components must a student engage in be considered completing it)?
What percentage of YCC students do you expect to complete the program?
C. PROGRAM ALTERNATIVES
Confirm the alternative programs or career pathways for similar students not in the YCC program.
How do these programs differ from the YCC program?
How do students know about them?
Confirm whether students not in the YCC programs can receive YCC or related services.
How easy is it for students in these programs to get YCC program components?
Can they take YCC classes?
Can they receive such learning opportunities as mentoring, job shadowing, transportation, and internships and if they can, how do these opportunities differ from the ones in YCC?
Can they receive the same support services as students in YCC?
Time: 2 hours
Staff involved: 2-5 individuals responsible for program oversight and recruitment and enrollment
The purpose of this section is to discuss how random assignment procedures might be integrated into existing program operations in a way that creates minimal disruption to program operations. The goal is to discuss the schools, programs, and grades to be included in the study; placement of random assignment within the processes diagrammed in part IIIB; students to be randomly assigned, distribution and collection of study forms, technology available for conducting random assignment, and notification of random assignment results.
A. SCHOOLS, PROGRAMS, AND GRADES FOR INCLUSION
Discuss which high schools and programs are candidates for random assignment based on their potential for oversubscription, a “clean” counterfactual, and a centralized application process
Should all high schools and programs be included in the study so that all staff will share equally in the evaluation? Discuss the pros and cons of including more schools and programs: smaller comparison groups from each school/ program but broader involvement.
Discuss specific grades to include in the evaluation.
Is it important that the study select the younger students who have a greater potential for being exposed to the full YCC experience? Or should all grade levels be included?
If students can enter the program in multiple grades, will problems arise because new students enroll in the program after random assignment but not the comparison sample? Will random assignment need to be conducted in each year in this case?
B. PLACEMENT OF RANDOM ASSIGNMENT
Discuss where in the application and enrollment process random assignment should be inserted. Use the flow chart of the application/enrollment process to pinpoint where random assignment might occur.
Where might random assignment occur? At the point of application? Eligibility determination? Enrollment? Other?
Should the point of random assignment vary across schools or programs?
Discuss when in the school year random assignment would occur. Use the timeline developed in IIIB to pinpoint timing. Does it matter whether random assignment occurs during the spring before freshmen year or the start of the school year?
Tradeoffs to consider in selecting the point of random assignment:
Is the point too early so that many students in the YCC group will not enroll in the program?
Is the point too early for staff to adequately determine program eligibility and program suitability?
Is this point too late because students are already committed to the program and it would be difficult to deny services at that point?
Is the point too late because program services may have already been received by applicants (such as assessments and case management)?
Does this point of random assignment make sense to staff in that it fits into normal program operations? Why or why not?
Discuss the specifics of random assignment.
Would it would be preferable to conduct it within occupation/industry groups (for example, separately for students who express interest in health occupations, technology occupations, and so on).
Would it need to occur on an individual basis over time or for a whole group at one time?
Discuss the aspects of the grantee’s existing procedures that might need to change to make random assignment occur smoothly.
C. STUDENTS TO BE RANDOMLY ASSIGNED
Discuss whether there needs to be any exemptions from random assignment.
Are there specific groups of students who are mandated to enroll?
Are there some students who are given preference for program enrollment (for example, siblings)?
Are there particular vulnerable populations of students where it would be preferable to select fewer comparison group members?
If we exclude these groups of students from the random assignment, do you think the resulting random assignment of YCC students would represent those in your program?
Discuss whether there should be a small number of “wild cards” that staff could use to allow a student into the program. If you had wild cards, what types of students might receive them?
D. DISTRIBUTION AND COLLECTION OF RANDOM ASSIGNMENT FORMS
Discuss how information about the study might be distributed to students and parents.
Are there existing outreach materials that can be amended to provide information on the study (such as brochures)?
Should new study materials be developed and distributed?
Will there be a way for students/parents to ask questions about the study?
Discuss how baseline and parent surveys and parent consent forms might be integrated into the current application and intake processes.
If the program uses application forms, can the study forms be inserted with the application forms?
If the program does not use application forms, how and when can the study forms be distributed to those interested in attending the program?
Who would collect these forms, ensure that they are filled out correctly, and send them to Mathematica for data entry?
How can students/parents ask questions about what it means to give consent to be in the study?
Discuss how study forms will be securely stored and sent to Mathematica for data entry.
Is there a secure location to store these materials?
If they are electronic, can they be electronically transmitted through a secured file transfer process?
Discuss the need to have the study forms translated into a language other than English and confirm what languages are needed.
E. COMPUTER SYSTEMS FOR RANDOM ASSIGNMENT
Discuss the technology that might be used to conduct random assignment.
If there is an existing lottery system in place, can we integrate this study’s random assignment processes into it and, if so, how can Mathematica obtain information on lottery results and updates reflecting changes off the wait list?
If there is not an existing lottery system, will you be able to use the PTS for random assignment?
If so, will the random assignment process be conducted separately at each study school or program or can it be done centrally in one place?
If not, what other data systems will you use?
F. NOTIFICATION OF RANDOM ASSIGNMENT RESULTS
Discuss how applicants and their families will be notified about the random assignment results.
Who will notify the students and their families?
Will they be notified as a group or individually?
G. MONITORING RANDOM ASSIGNMENT
Explain that staff will need to use the PTS to ensure that comparison group students do not receive YCC services for the duration of the grant.
Will all staff with responsibility for assigning students to YCC program components have access to the PTS?
Will this “monitoring” process be centralized or should it be performed separately in each school or program? And by whom?
What other procedures can be employed to ensure that comparison group members do not receive YCC services?
Can we provide information to counselors to prohibit comparison students to enroll in YCC courses, undertake work-based learning activities, or receive support services?
Can we use sign in logs for activities to monitor who is receiving services?
What other procedures can be used to ensure comparison students are not inadvertently provided YCC services?
Time: 1 hour
Staff involved: 2-5 individuals responsible for program oversight and recruitment and enrollment
The purpose of this section is to address the staff that needs to be involved in study activities, the approvals that are needed from any oversight or review boards, and the timeline and next steps.
A. STAFFING AND TRAINING
Explain that we need to be able to have regular communication with staff about the study, particularly during the random assignment stage.
Who will be the grantee’s primary contact with the evaluation team? Can we have their contact information?
Will other individuals need to be contacted when issues arise (for example, when we have questions about random assignment, monitoring of comparison group activities, or notifying students about something)? Can we have their contact information?
What is the best way to communicate with individuals (email, phone)? What is the best time of the day to contact them?
How frequently do you think your Mathematica/SPR team liaison needs to meet with this contact to continue finalizing arrangements for implementing random assignment?
Explain that we will be conducting training about random assignment and monitoring.
What staff members will conduct random assignment and ensure the comparison group students do not receive services?
What staff members need to be trained on doing random assignment procedures?
When and where might the training on random assignment procedures take place?
Can these staff participate in training using technology? Attend a Go To meeting?
Can you download software on your machine in order to view a training session?
B. OVERSIGHT AND REVIEW
Explain that districts/schools/states often have rules that govern research and that we want to make sure that our research complies with these rules and accompanying procedures.
Who administers and provides oversight and management to the program? Is there a governing board, advisory group or oversight bodies?
What is the process by which we engage the administrators or governing bodies about the evaluation? How long does it generally take to get approval?
What are the rules governing parent consent in your school/district/state?
What procedures are in place for reviewing this research, for example an Institutional Review Board (IRB) or Board of Education?
Who would we want to contact about finding out whether such a body exists? Do you have their contact information?
If there is an IRB or similar research approval body, what is their policy on accepting approvals from other IRBs?
Do you anticipate any opposition from any of these bodies or administrators? If so, please describe potential opposition and how we might overcome it.
C. TIMELINE OF NEXT STEPS FOR WORKING WITH YOU
Decide on final sites included in the evaluation will occur shortly after all site visits have occurred
Work with your site to further customize of study procedures to your site
Provide random assignment training at a time convenient for you
Work with you to begin random assignment of students
Monitor random assignment throughout the duration of the grant
Time: 30 minutes
Staff involved: 2-5 individuals responsible for program oversight and recruitment and enrollment
The purpose of this section is to address any remaining concerns and to answer any remaining questions that staff might have and to ensure that we have thoroughly worked with the site to develop random assignment procedures that are least burdensome.
Determine if they see any other potential obstacles to conducting random assignment that we need to address.
Ask if they have questions or concerns about random assignment or the evaluation that we have not covered.
A.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Sarah Thomason |
| File Modified | 0000-00-00 |
| File Created | 2021-01-28 |