Memo - Nonsubstantive change request: Updates to Semi-structured interview and Observation Checklist

MEMO

To: Steph Tatham; Office of Information and Regulatory Affairs (OIRA), Office of Management and Budget (OMB)

From: Samantha Illangasekare, Office of Planning, Research and Evaluation (OPRE), Administration for Children and Families (ACF)

Re: Nonsubstantive change request for revisions to interview and program observation protocols for the YEARS information collection (OMB # 0970-0470)

Date: October 31, 2016

Based on feedback from additional trainings and data collection, we have made a few minor updates to the Youth Education and Relationship Services (YEARS) interview and program observation protocols (approved under OMB #0970-0470). This memo outlines the proposed revisions and the corresponding rationale for the changes. The proposed changes do not increase participant burden and can be considered administrative/non-substantive changes. Tracked and clean versions of the revised protocols are attached.

Proposed revisions to interview protocol

Revisions to Introduction and Conclusion sections

• We have provided more details in the Introduction (Section 1) and Conclusion (Section 6) sections of the interview protocol to describe the study and the use of the data.

  • Rationale: During the recruitment and data collection phases, participants have raised several questions about the study and how the information will be used. We have added language to address these questions, which will allow the interview teams to be consistent in terms of how they respond to requests for additional information about the privacy of the data, the use of the data, and how findings will be reported.

Revisions to protocol instructions

• We have revised the instructions throughout the protocol to provide more guidance to interviewers about the subject of certain questions and what questions to repeat when discussing multiple programs or multiple partners.

  • Rationale: The revised instructions are meant to help streamline the interview questions, avoid unnecessary repetition when asking about multiple programs, and avoid confusion when specifying whether a question should be asked about the grantee organization, the HMRE program being implemented, and/or a partner organization.

Revisions to probes

• We have spelled out acronyms in certain sections of the protocol, and provided examples to provide more guidance to interviewers. For example, in the Youth Characteristics section (Section 2,) one probe asks about the sexual orientation of the majority of the youth served in the program. Included was: “…LGB (lesbian, gay, or bisexual)…” This was amended to be “…lesbian, gay, or bisexual…,” without the acronym.

In another example, in Section 4, Question #4 we added in examples of different types of facilitation styles.

• Rationale: In additional trainings, team members agreed that when asking about sexual orientation it made more sense to not include the “LGB” acronym. We provided examples of facilitation styles in order to increase the consistency of data collection across interviews. Additionally, this change will allow staff to discuss the specific facilitation styles being observed through program observations.

Proposed revisions to observation protocol

Guidance for how to define/operationalize items and response categories

• Throughout the protocol, we have provided specific definitions and guidance for how to interpret sections of the protocol. For example, in Section C, we defined the difference between a “Major Focus” and “Minor Focus” on specific Healthy Marriage and Relationship Education topics during the observed program session.

  • Rationale: After completing several site visits, staff noted sections of the protocol, like Section C, where more clarity was needed to improve the use of the observation tool and consistency in the data being collected.

• We have provided additional guidance on how to interpret the response categories for select items. For example, we have noted in several places where a frequency scale ranging from never to always should be considered. We also recommend revising the frequency scale categories from never, rarely, sometimes, always to never, rarely, often, always.

• Rationale: For implementation practices that could occur multiple times during a session, a frequency scale is helpful to assist with coding the activity more accurately. Also, staff provided feedback that it was difficult to distinguish between “rarely” and “sometimes” on the original frequency scale, thus the proposed revision to replace “sometimes” with “often.”

• We have removed the N/A response option from select sections of the protocol.

  • Rationale: The purpose of this change was to contribute to consistency in the data collected from the observations. For certain implementation practices that could definitely be observed (e.g., require a yes/no response and were always applicable), the team decided that N/A was not a necessary response option. Removing N/A as an option will help clarify whether “no” or “strongly disagree” is the appropriate answer for something a staff member did not observe, rather than reporting N/A.

Additional instructions for observers

• In some sections of the protocol we have provided additional notes and instructions to assist site visit teams in accurately coding the observations. For example, we added a note to item E17 on the use of technology and/or social media requesting that staff provide details about what technology was used and how it was used, if this practice is observed. In section G, we have added a note about the difference between “inclusive language” and “affirmative language” to better define these response categories.

  • Rationale: These notes were added to the observation protocol to prompt observers to provide more detail in certain places and to clarify protocol questions and response categories.

2