National Center of Excellence for Infant and Early Childhood Mental Health Consultation
Supporting Statement
1. Respondent Universe and Sampling Methods
To assess reach, implementation and impact of the IECMHC program, the IECMHC Evaluator will elicit key data from two groups of respondents: a) mental health consultants and other target audiences, and b) State/Tribal representatives. Expert mentors will be selected by the IECMHC workgroup to develop and implement an IECMHC Strategic Plan in 10 pilot States and 1 American Indian/Alaska Native pilot Tribe for the main program, and 4 pilot States for the home visiting component. Strategic plans will be based on each State’s and Tribe’s needs, resources and vision for building an IECMHC system. Services supplied by the mentors (with support from the workgroup) will include individualized monthly training and technical assistance (TTA), which will be informed by the Toolkit. Mentors will locate mental health consultants and other target audiences through marketing and other outreach efforts. Typical audiences for IECMHC activities will also include State and Tribal early childhood leadership (e.g., maternal and child health, behavioral health, child care, Head Start, education and child welfare agencies); academic institutions and professional organizations involved in the training of early childhood mental health professionals; researchers; private foundations and other private sector partners that seek to promote and support IECMHC; and community-based organizations that want to build IECMHC into the repertoire of services they provide.
Below their descriptions is a table that outlines which instruments participants will be asked to complete.
Mental Health Consultants and Other Target Audiences. These individuals represent the target audience for Toolkit-informed trainings and associated TTA. All recipients of the webinars and TTA will be asked to provide feedback on the quality and applicability of the trainings through online pre-assessment, feedback and 2-month follow-up surveys on training and technical assistance they received (as noted in Table 4 below); no sampling will occur. While participation in all surveys will be voluntary, the IECMHC Evaluator expects all participants to complete pre-assessments (which is a condition of webinar registration) and feedback immediately following the training events, based on past experience with similar audiences and training content. Based on the response rates garnered for past projects with similar audiences, the IECMHC Evaluator expects 75% of recipients to return the follow-up surveys for the trainings and technical assistance.
State/Tribe Representatives. The 15-17 representatives who will be chosen to work with the IECMHC mentors will be asked to complete the annual Cumulative Training and Technical Assistance Service Assessment that will be based on their impressions of the reach and impact of the IECMHC program for each Option Year. Respondents will be asked to report on which services they received from IECMHC, whether the services led to improvements in their capacity to train mental health consultants and improve infant and early childhood programs, and their level of satisfaction with the services. The assessment will also capture the number of centers and programs (including Head Start, home visiting and other preschool programs) that have begun using IECMHC as a result of the CoE’s work.
In addition, beginning in Option Year 1, the mentors will work with representatives (approximately 1 per pilot site, or 15-17 representatives) to collect and report on the progress of States/Tribes to achieve program implementation and outcome benchmarks. Process measures include scope of IECMHC implementation across the State/Tribe and the development of IECMHC-related marketing materials, training standards, program evaluation and sustainable financing. Outcome measures include year-over-year expulsion rates from childcare and pre-K programs, and (for those receiving MIECHV funding) selected MIECHV (Maternal Infant and Early Childhood Home Visiting) measures/constructs regarding maternal and newborn health, school readiness and achievement, and coordination and referrals for other community resources and supports. Collection and reporting of process measures will occur quarterly, then will be summarized in the annual report. Collection and reporting on the outcome measures will occur annually.
If States/Tribes are not already collecting this data, the mentors will work with the representatives to set up appropriate mechanisms for capturing the information. Mentors will use the IECMHC Annual and Quarterly Benchmark Data Collection Forms that have been developed by the IECMHC evaluation team to collect and report State/Tribe information. As engaging in these annual assessments and reporting is a condition of their involvement in the IECMHC program, the IECMHC Evaluator expects 100% of State/Tribe representatives to supply this data.
TABLE 4
Survey |
Collection Method |
Respondents |
Service Pre-Assessment Form |
Web-based form |
150 mental health consultants and other stakeholders (100% of participants) |
Training Feedback Form |
Web-based form |
112 mental health consultants and other stakeholders (75% of participants) |
Training Follow-up Form |
Web-based form |
112 mental health consultants and other stakeholders (75% of participants) |
Technical Assistance Follow-up Form |
Web-based form |
30 mental health consultants and other stakeholders (75% of participants) |
IECMHC Cumulative Services Assessment Form |
Web-based form |
17 State/Tribal representatives (100% of participants) |
IECMHC Annual and Quarterly Benchmark Data Collection Forms |
Web-based form, in consultation with State/Tribe representatives |
17 State/Tribal representatives (100% of participants) |
2. Information Collection Procedures
Mental Health Consultants and Other Target Audiences. As indicated in Table 4 above, all pre-assessment, feedback and follow-up surveys for mental health consultants and other target groups described above will be administered electronically. Before each of the 5 IECMHC webinars, the link to the pre-assessment survey and registration materials will be sent in an email to mental health consultants and others. Participants will be given 2 weeks to respond, and informed of the closing date. During the 2-week window, the pre-assessment can be accessed at any time by these audiences. After the 2-week window has lapsed, the survey will close. After the webinars and TTA events, participants will receive a link to the feedback form by email. They will be given 2 weeks to respond, and informed of the closing date. During the 2-week window, the feedback form can be accessed at any time by these audiences. After the 2-week window has lapsed, the survey will close. Two months following the webinars and TTA events (including individualized TA), participants will receive a link to the follow-up training and TA surveys by email. Participants will be given 4 weeks to respond, and informed of the closing date. During the 4-week window, the follow-up form can be accessed at any time by these audiences. After the 4-week window has lapsed, the survey will close.
State/Tribe Representatives. Approximately 1 month before the close of the Option Year, State/Tribe representatives will receive a link to the cumulative assessment by email, and given 4 weeks to respond. Representatives will be informed of the closing date. During the four-week window, they will be able to access the cumulative assessment at any time. After the 4-week window has lapsed, the assessment will close.
Mentors will have some discretion about how they will collect the above-described quarterly and annual program data from State/Tribe representatives. They can communicate via email, telephone call or face to face. After consulting with their representatives, mentors will enter the data into an online instrument (which they will receive as a link in an email at least 4 weeks prior to the reporting deadline). They can complete the entire form themselves based on information supplied by the States/Tribes, or ask the representatives to fill out portions if appropriate. State-level reporting is due at the end of each quarter (beginning in Option Year 1) for quarterly data, and at the end of each Option Year for annual data, and mentors will be informed of these due dates when they receive the links to the instruments.
3. Methods to Maximize Response Rates
Mental Health Consultants and Other Target Audiences. Anticipating that participating in training events is in the self-interest of prospective service recipients, and based on experience with similar populations and training content, the IECMHC Evaluator expects the response rate for the Service Pre-Assessment form to be 100% because participants must complete it in order to register for the national webinar or virtual TA session. Given the simplicity of the forms and plans for administration at the end of the webinars and TA events, the IECMHC Evaluator anticipates close to a 75% response rate for the Training Feedback form.
To ensure that target audiences are aware of upcoming training offerings and to maximize response rates, prospective participants will receive an email notification 3 weeks ahead of the planned events, which will inform them of the nature of the event and ask them to watch their inboxes for registration details. After the link to the registration and pre-assessment has been sent, recipients will receive up to 3 reminder emails (which will also include the link) 3, 7 and 10 days after the initial email. Each of the reminder emails will include the link to the registration and pre-assessment, and the deadline for responding.
The feedback form will be sent to participants immediately following the training event, and email reminders will be sent to nonrespondents 3, 7 and 10 days later. Each of the reminder emails will include the link to the form and the deadline for responding.
The training and/or technical assistance follow-up form will be sent to participants approximately 2 months after the training event, and email reminders will be sent to nonrespondents 1, 2 and 3 weeks later. Each of the reminder emails will include the link to the form and the deadline for responding.
State/Tribe Representatives. It is expected that reporting from State/Tribe representatives will have a 100% response rate because this data collection is integrated into the existing required progress reporting system stipulated in the IECMHC statement of work. Nevertheless, following the initial notification that reporting is due, reminders will be emailed to nonrespondents 1, 2 and 3 weeks later.
In addition to meeting contractual obligations for data collection and reporting, the content and methods of the proposed forms were informed by a review of relevant assessment literature, collaboration with SAMHSA experts, and by the experience of evaluation principal investigators, all of whom have substantive experience in designing and conducting assessments, including assessment and evaluation of similar training and technical assistance centers. The forms and proposed methods have been shown to work well for this type of assessment activity. If necessary, the IECMHC evaluation team will conduct focus groups and/or pretest items/forms involving 9 or fewer participants to identify views and concerns of customers.
The IECMHC Evaluator has full responsibility for the development of the overall design and assumes oversight responsibility for data collection for the IECMHC program. The following individual is primarily responsible for overseeing data collection and analysis:
Bertha M. Gorham, Ph.D.
Evaluator
IECMHC Program
(919) 870-5808
(919) 618-3456 (Cell)
The following individuals serve as statistical consultants to this project:
Deborah Perry, Ph.D.
Associate Professor, Center for Child and Human Development
Georgetown University
Katharine Sinclair
Research Assistant
Education Development Center, Inc.
KSinclair@edc.org
Audrey Adade, MSSW
Contracting Officer’s Representative (COR)
Public Health Advisor
Mental Health Promotion Branch
Division of Prevention, Traumatic Stress and Special Programs
SAMHSA/ Center for Mental Health Services
U.S. Department of Health & Human Services
(240) 276-2204
Maureen Madison, Ph.D.
Alternate Contracting Officer’s Representative (ACOR)
Public Health Advisor
Division of Prevention, Traumatic Stress and Special Programs (DPTSSP)
Center for Mental Health Services (CMHS)
Substance Abuse and Mental Health Services Administration (SAMHSA)
Maureen.Madison@samhsa.hhs.gov
(240) 276-1772
LIST OF ATTACHMENTS
Attachment A Service Pre-Assessment Form
Attachment B Training Feedback Form
Attachment C Training Follow-up Form
Attachment D Technical Assistance Follow-up Form
Attachment E IECMHC Assessment of Cumulative Toolkit-related Services
Attachment F IECMHC Annual and Quarterly Benchmark Data Collection Forms
Page
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | natalie.j.henrich |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |