Part B Supporting Statement Final_080218

Supporting Statement for the Evaluation of the Poetry Out Loud 
Program, Part B 

Table of Contents
Part B. Collections of Information Employing Statistical Methods ......................................... 1 

Table of Attachments
ATTACHMENT A: CONSENT AND ASSENT FORMS
ATTACHMENT B: PRINCIPAL AND SCHOOL DISTRICT SUPERINTENDENT
LETTER OF SUPPORT
ATTACHMENT C: DATA SHARING AGREEMENT
ATTACHMENT D: PRE- POST- SURVEY
ATTACHMENT E: POL STUDENT INTERVIEW PROTOCOL
ATTACHMENT F: POL TEACHER INTERVIEW PROTOCOL
ATTACHMENT G: STATE ARTS AGENCY ADMINISTRATOR INTERVIEW
PROTOCOL
ATTACHMENT H: STUDENT FOCUS GROUP PROTOCOL
ATTACHMENT I: COGNITIVE TESTING REPORT

Part B. Collections of Information Employing Statistical Methods
Part B applies to data collection employing statistical methods only. For this study, data
collection employing statistical methods includes pre- and post-student surveys of Poetry Out
Loud (POL) program participants and non-participants in the 10 schools.
1. Describe (including a numerical estimate) the potential respondent universe and any
sampling or other respondent selection methods to be used. Data on the number of
entities (e.g., establishments, State and local government units, households, or persons)
in the universe covered by the collection and in the corresponding sample are to be
provided in tabular form for the universe as a whole and for each of the strata in the
proposed sample. Indicate expected response rates for the collection as a whole. If the
collection had been conducted previously, include the actual response rate achieved
during the last collection.
This section outlines the selection criteria that defines the sample for the study and describes
the potential respondent universe and anticipated response rates. To select the study’s sample,
the research team will recruit ten (10) schools from a pool of approximately 2,300 participating
schools that meet four (4) selection criteria: (1) states are optimally implementing Poetry Out

1

Loud;1 (2) schools are implementing mandatory POL programming in at least one grade level;2
(3) schools meet the necessary conditions to implement the study, including having a minimum
of 900 POL-participating students and about 900 non-participants who are matched using
propensity score methods, allowing the implementation of a school-wide online survey, and
having the ability to provide student-level data for all students in the school; and (4) schools
possess other features so as to achieve a good mix of school sites primarily in terms of
geography, and secondarily in terms of locale (urban/rural) and student body composition. After
the identification of 18 states that were optimally implementing POL, the NEA reviewed past
documentation and reports shared by State Arts Agencies for evidence of schools that might
meet the study’s selection criteria. The NEA and/or its contractor then reached out to POL
coordinators in State Arts Agencies in selected states for individualized follow-up about
identified schools. Through this process, NEA learned about potential schools that might be
eligible to participate in the study. The contractor then conducted individualized follow-up with
school principals to learn more about the school’s history with POL and to informally assess the

1

As noted in Part A, schools will be selected from states that are optimally implementing POL. Optimal conditions
as determined by the Poetry Out Loud program partners are as follows: states should have an overall count of
participating students exceeding 2,500; an overall count of participating schools exceeding 20; presence of ancillary
activities supporting state finals competitions, direct student exposure to a working artist, and celebratory activities
for students and families such as a welcome banquet or reception; formal teacher recognition at the state level;
opportunities for winning students to perform at local arts events throughout the state; strong support for the POL
program from executive leadership at the state arts agency; workshops for teachers and/or students facilitated by the
state arts agency; matching or overmatching of POL grant money with funds from the state arts agency; and an
annual program assessment. Eighteen states were identified by the NEA and the Poetry Foundation as optimally
implementing POL.
2
NEA defined “mandatory” participation at the classroom level as individual teachers deciding that their class will
participate in POL and that every student in the class will be required to select and memorize a poem and compete in
the classroom and/or school competition. Mandatory participation at the grade level is when all teachers in a
particular grade or grades agree to participate in POL and require all students in that grade level to select and
memorize a poem and compete in the classroom and/or school competition. Selecting schools with mandatory
participation prevents self-selection bias in the sample.

2

capability of the school to meet the study requirements, including, but not limited to, the use of
mandatory participation for POL implementation.
Within the selected schools – selected from the criteria listed above, including mandatory
participation in POL at the classroom or grade level - a maximum of 900 students participating in
POL and 900 students not participating in POL will be identified to respond to the surveys. All
students in the selected schools will be asked to complete the survey. Because mandatory
participation is based on either a student’s English Language Arts teacher or grade level, the
research team will be able to identify POL versus non-POL participants by either teacher or
grade.
As shown in Exhibit 1, the research team’s goal is to achieve a response rate of 80% for the
baseline surveys, which will occur prior to the implementation of Poetry Out Loud content in the
classrooms. We chose an 80% response rate as a goal for the survey following OMB guidelines,
the threshold where potential biases are acceptably small. To achieve this goal, we put in place
specific strategies to help boost response rates. First, we designated a POL liaison at each of the
schools to support survey data collection. POL liaisons are experienced English Language Arts
teachers at the schools selected for the study. We expect that they will play an important role in
relaying the importance of the study and helping us obtain the support from other teachers at the
school. Second, with the support of teachers, we will be asking that students complete the
surveys at school. Allotting a specific time to fill out the survey will help response rates. Lastly,
we designed the survey keeping in mind the length and the ease for respondents to answer, both
of which can lead to improved response rates. To avoid survey fatigue, we kept the length of the
survey to about 15 minutes. We also used open ended items sparingly and spread throughout the
survey to reduce the cognitive burden associated with responding to these kinds of questions.
3

If survey response rates fall below 80%, as we expect it will happen in the post survey, we
have plans to conduct missing data analyses and correct for non-response bias, as further
explained in section B.3. For the post-survey, the research team will administer the survey to
respondents of the baseline survey and anticipates a lower response rate of approximately 50%
due to attrition and other unexpected events, since the data collection will occur about nine
months after the baseline survey.
Exhibit 1: Data Collection to be Analyzed Using Statistical Methods
Data Source
Youth Baseline/
Pre-Survey:
Youth Baseline/
Pre-Survey:
Youth Followup/Post-Survey
Youth Followup/Post-Survey

Respondents
POL
Participants
NonParticipants
POL
Participants
NonParticipants

Timing of Data Collection
Prior to start of POL curriculum (est.
Sept. 2018)
Prior to start of POL curriculum (est.
Sept. 2018)
After conclusion of POL (est. June
2019)
After conclusion of POL (est. June
2019)

Response
Universe

Estimated
Response Rate

9,000

80%

9,000

80%

7,200

50%

7,200

50%

2. Describe the procedures for the collection of information, including:






Statistical methodology for stratification and sample selection.
Estimation procedure.
Degree of accuracy needed for the purpose described in the justification.
Unusual problems requiring specialized sampling procedures,
Any use of periodic (less frequent than annual) data collection cycles to reduce
burden.

The purpose of the evaluation of the Poetry Out Loud program is to understand studentlevel outcomes associated with the implementation of POL programs. The evaluation is mixed
method, combining a quasi-experimental design involving a treatment group of students
participating in POL and a comparison group of non-participating students from the same
schools. The quasi-experimental design will include pre- and post-student surveys for the
treatment and control groups, analysis of student record data for all students (treatment and
comparison), coupled with qualitative on-site data collection to help understand POL program

4

implementation and the counterfactual (i.e., the experiences of those in the comparison group).
This design will allow the research team to analyze all outcomes of interest. It also helps us to
provide insight into the factors affecting those outcomes and to identify how outcomes have
changed after implementation of the program.
To learn about the efficacy of the Poetry Out Loud program, SPR will select a purposive
sample of 10 POL-participating schools across the U.S. to conduct quantitative and qualitative
data collection activities. In consultation with the NEA, SAA staff, and other project partners,
SPR will recruit school sites that meet the criteria to be part of the study. Specific details about
school site selection are addressed in detail in the section that follows.
As noted in the evaluation planning matrices, the study is guided by a series of research
questions focused on the assessment of the program’s impact in three different domains:
students’ academic engagement and performance, poetry engagement and appreciation, and
socio-emotional development. Regarding student record data, SPR will be asking for the sample
universe – that is, all de-identified student records at the school. Regarding the student survey,
SPR will request that all students in the school fill out the pre- and post-survey. Regarding onsite qualitative data collection, SPR will work with the school and teachers to select classrooms
to visit and individual students to interview and invite to participate in focus groups. For
selecting teachers for interviews, SPR will work with the POL liaison to identify how many
teachers are participating in POL in a given school. We will then seek teachers who are willing
to participate in an interview with SPR and whose schedules align with the researcher’s
availability while on site. If POL participation is spread across multiple grade levels, SPR will
try to interview participating teachers at those different grade levels. For students, SPR will
provide the study liaison with parameters for teachers in recommending students for interviews.
5

Because this study is focused on POL in optimally implemented conditions, the parameters
include students who are doing fairly well in school. Another reason to select “good” students is
so that we do not take up class time for any students who are struggling academically. In
addition, the study team will be seeking a broad diversity of student interviewees – in terms of
grade level, rural/urban residents, race/ethnicity, and gender – across all 10 schools and will
suggest appropriate school-level diversity through our parameters.
Regarding the degree of accuracy needed for the purpose described in the justification,
the results of our power analyses utilizing 1,800 students per school yield minimum detectable
effects (MDEs) for a pooled sample of 18,000 (1,800 for each of the 10 schools) on some of the
outcomes of interest, which are as follows: 1) Based on the results from Crombie, Walsh, and
Trinneer’s (2003) study examining the effects of a similar program to POL on students’
confidence (M=3.86, SD=.88), we estimated a minimum detectable effect of .06 for the presurvey and .08 for the post-survey, assuming a 50% response rate and using a standard level of
power (80%) at the 90% confidence level, and 2) Following the same assumptions and using the
results of a study examining the impact of a theater intervention on reading scores (M=193.16,
SD=.22.74), (Inoa, Weltsek, & Tabone, 2014), we estimated a minimum detectable effect of
1.37.
To build the comparison group that is similar to POL participants, propensity score
matching will be used to construct a comparison group that is most similar to the group that
participates in POL programming at least on observable characteristics. Recall that schools
selected to participate in the study will be schools that mandate participation in POL at either the
classroom or at the grade level (see above for a more detailed explanation). A set of covariates
will be used to estimate the propensity score. The selection of covariates will be based on
6

previous research examining the relationships between variables of interests (e.g., age, gender,
race/ethnic background, English Learner status, prior academic achievement). Data of POL and
non-POL participants will be pooled to estimate the propensity score [(Pr(X) = Pr(T=1|X)] for
each subject. To estimate the propensity score for each subject, logit regression will be used,
with POL participation as the dependent measure, and a range of demographic and other
characteristics as independent measures to establish the relative weights for each of the
independent measures in “predicting” POL participation. The next step is to match each student
in the group of students who participated in POL to another individual student in group that did
not participate in the program. To do this, we will use the “nearest neighbor” approach in the
selection process, meaning that we will select the comparison group member whose propensity
score is closest to the respective POL participant. We also plan to use replacement, so that a
potential comparison group member can be matched to several POL participants. Lastly, we will
assess the matching and perform sensitivity tests to assess whether other approaches would be
preferable before estimating the average POL participation effect on student outcomes.3
The primary means of reducing burden associated with data collection will be that we
will be requesting student record data that schools already collect.
3. Describe methods to maximize response rates and to deal with issues of nonresponse.
The accuracy and reliability of information collected must be shown to be adequate for
intended uses. For collections based on sampling, a special justification must be
provided for any collection that will not yield "reliable" data that can be generalized to
the universe studied.

3

Other matching methods include caliper and radius matching, stratification/interval matching, or kernel matching.

7

The research team will work to maximize student survey response rates with multiple
strategies. First, in order to be included in the sample of 10 schools, schools will need to agree to
request that all students in the school to take the pre- and post-survey, and to encourage
participation, although students will have the option to decline to take the survey. As discussed
in Supporting Statement A, students will receive individual invitation links via email from SPR.
SPR will therefore begin the survey administration process by collecting all student emails from
each school. SPR will then input these emails into the survey administration platform
(SurveyGizmo) using their Email Campaign feature linked with a simple mail transfer protocol
(SMTP). By sending SurveyGizmo emails through this method, each student will receive an
email containing a unique link to take the survey. The research team will work with the POL
liaison to determine the best way to administer the survey at each school; however, we will
suggest that each teacher set aside class time for students to take the survey either via a
classroom set of computers or in a computer lab. To incentivize participation, the NEA will offer
to brief leaders of participating schools on study results after the conclusion of the study. After a
school has agreed to this condition of participation, the research team will work with the school
to coordinate survey administration through the school, at the classroom level, and through
follow up correspondence with participants. The follow up correspondence will occur through
monitoring response rates via our SurveyGizmo platform. Through SurveyGizmo’s Email
Campaign feature, SPR will be able to email individual student reminders to those who have not
completed the survey while keeping their information confidential to the research team. In
addition to securing school administration-level buy-in for supporting full student participation
in the pre and post surveys, SPR will also work with one or more POL coordinator teacher(s) to
oversee survey administration and to encourage high rates of student response. To address

8

challenges to data collection and the danger of lower than expected response rates, SPR will
undertake some of the following strategies to address low response rates:
(1) communicate with participating sites to better adapt our data collection strategies to
specific sites;
(2) identify an evaluation point of contact at each site who could help administer the
surveys. Because the survey is online, our POL liaison will help administer the survey
through alerting teachers to the survey launch and coordinating classroom times for
students to take the survey;
(3) provide a stipend and prepare a standardized training for evaluation points of contact
to support data collection;
(4) track the completion of online surveys by site in order to conduct appropriate followup to encourage survey completion. We will specifically monitor the number of
respondents that are POL participants and non-POL participants and target follow-ups to
ensure that we have an equal balance of both groups; and
(5) actively collaborate with the site evaluation point of contact leading up to and during
the administration and return of student baseline and follow-up surveys.
We will conduct a non-response bias analysis to determine the impact of non-response.
To do this we will compare the characteristics of those who responded to the survey with the
pool of program participants on various demographic characteristics (e.g., age, grade, gender,
race and ethnicity). The pool of participants will be obtained from the student administrative data
we receive from schools. Through this comparison, we will determine whether there are
statistically significant differences between the actual and potential survey respondents.
Depending on the results, we will determine if there is need to address non-response using
additional statistical procedures such as weighting.
Administrative data are typically available for the vast majority of students since schools
have to collect these data routinely to meet federal and state accountability requirements.

9

Nevertheless, the research team will assess the patterns of missing data and determine whether it
is necessary to correct for missing data using other methods.
Describe any tests of procedures or methods to be undertaken. Testing is
encouraged as an effective means of refining collections of information to minimize burden
and improve utility. Tests must be approved if they call for answers to identical questions
from 10 or more respondents. A proposed test or set of test may be submitted for approval
separately or in combination with the main collection of information.
The research team tested the survey questionnaire conducting four cognitive interviews
with high school students in October - November 2017. The objectives were to: (a) detect
questionnaire design problems; (b) check for students’ interpretation of the questions and the
reasoning behind their answers for question items; (c) detect confusing wording; (c) ensure
questionnaire flow; and (d) estimate average time to complete the survey. Changes were made to
the survey instrument following completion of cognitive testing. The Cognitive Testing Report
can be found in Attachment I.
4. Provide the name and telephone number of individuals consulted on statistical aspects
of the design and the name of the agency unit, contractors, grantees, or other person(s)
who will actually collect or analyze the information for the agency.

Name

Organizational Affiliation
and Address

Title (Project Role)

Phone Number

Parties doing the data collection
Melissa Mack

Project Director, CoPrincipal Investigator

Social Policy Research
1333 Broadway, Suite 310
Oakland, CA 94612

10

(510) 788-2478

Name

Organizational Affiliation
and Address

Title (Project Role)

Phone Number

Renatta DeFever

Quantitative Task Lead

Social Policy Research
1333 Broadway, Suite 310
Oakland, CA 94612

(510) 788-2459

Rachel Estrella

Co-Principal Investigator

Social Policy Research
1333 Broadway, Suite 310
Oakland, CA 94612

(510) 788-2481

(202) 682-5535

NEA and Poetry Foundation staff consulted
Patricia Moore
Shaffer

Deputy Director | Research
and Analysis

National Endowment for the Arts
400 7th Street SW | Washington DC
20506

Melissa Menzer

Program Analyst | Research
& Analysis

National Endowment for the Arts
400 7th Street SW | Washington DC
20506

Lauren Miller

Program Manager |
Literature & Arts Education
Division

National Endowment for the Arts
400 7th Street SW | Washington DC
20506

Eleanor Billington

Division Coordinator |
Literature & Arts Education
Division

Andi Mathis

State & Regional Specialist
| Partnership

National Endowment for the Arts
400 7th Street SW | Washington DC
20506

Justine Haka

Program Associate
Programming and Events

Poetry Foundation
61 West Superior Street, Chicago,
IL 60654

312-787-7070

School of the Arts, Virginia
Commonwealth University
325 N Harrison Street, Rm 201
Richmond, VA 23284

804-828-6875

National Guild for Community Arts
Education
Data Collection

212.268.3337 x15

University of California at Davis,
School of Education
One Shields Avenue
Davis, CA 95616-5270

(530) 754-9150

National Endowment for the Arts
400 7th Street SW | Washington DC
20506

202-682-5548

202-682-5490

202-682-5001

202-682-5430

Contractor’s technical working group consulted
Sarah Cunningham

Executive Director for
Research
Director, Arts Research
Institute

Jonathan Herman

Executive Director

Jamal Abedi

Professor

11

Name
Philip de Sa e Silva

Title (Project Role)
Upper School English
Teacher

Organizational Affiliation
and Address
St. Paul Academy and Summit
School
1712 Randolph Avenue
St. Paul, MN 55105

Phone Number
651-698-2451

Derek Fenner

Arts Learning Program
Manager

Alameda County Office of
Education, Oakland, California

(510) 670-7730

Andrea Santos

Teacher
2016 West Virginia Teacher
of the Year
Fine Arts Department Chair

Logan High School, West Virginia

(304) 946-2444

State Arts Agency Administrators consulted
Josy Miller

State Arts Agency
Administrator

Emily Reece

State Arts Agency
Administrator

Diane Daily

State Arts Agency
Administrator

Melissa Wray

State Arts Agency
Administrator

Virginia Sanders

State Arts Agency
Administrator

Charlotte Smelser

State Arts Agency
Administrator

Monica Smith
Grable

State Arts Agency
Administrator

Julianne Gadoury

State Arts Agency
Administrator

Kay Potucek

State Arts Agency
Administrator

Maryjane
Dorofachuk

State Arts Agency
Administrator

California Arts Council
1300 I Street, Suite 930
Sacramento, CA 95814
Georgia Council for the Arts
75 Fifth Street, NW, Suite 1200
Atlanta, GA 30308
Massachusetts Cultural Council
10 St. James Avenue, 3rd Floor
Boston, MA 02116-3803
The Loft Literary Center
Suite 200, Open Book Building
1011 Washington Avenue South
Minneapolis, MN 55415
Missouri Arts Council
815 Olive Street, Suite 16
St. Louis, MO 63101-1503
Mississippi Arts Commission
Suite 1101A, Woolfolk Building
Jackson, MS 39201
Montana Arts Council
830 N. Warren Street
Helena, MT 59620
New Hampshire State Council on
the Arts
19 Pillsbury Street, 1st Floor
Concord, NH 03301-3570
New Jersey State Council on the
Arts
33 West State Street,
4th Floor
Trenton, NJ 08608
Nevada Arts Council
716 North Carson Street, Suite A
Carson City, NV 89701

12

916-322-6385
404-814-4017
617-727-3668, ext.
262
612-215-2590

314-340-6845
601-359-6529
406-444-6522

603-271-0791

609-292-4609

702-486-3738

Jade Triton

State Arts Agency
Administrator

Chiquita Mullins
Lee

State Arts Agency
Administrator

Gayle G. Cluck

State Arts Agency
Administrator

Meredith Callis

State Arts Agency
Administrator

Anina Moore

State Arts Agency
Administrator

Casey Polczynski

State Arts Agency
Administrator

Lisa Jaret

State Arts Agency
Administrator

Jim Wolfe

State Arts Agency
Administrator

Teachers & Writers Collaborative
540 President Street, 3rd Floor
Brooklyn, NY 11215
Ohio Arts Council
Rhodes State Office Tower
30 East Broad Street, 33rd Floor
Columbus, OH 43215-3414
Pennsylvania Arts Council
216 Finance Building
Harrisburg, PA 17120
Tennessee Arts Commission
401 Charlotte Avenue
Nashville, TN 37243
Texas Commission on the Arts
E. O. Thompson Office Building
920 Colorado, Suite 501
Austin, TX 78701
Virginia Commission for the Arts
Main Street Centre
600 East Main Street, Suite 330
Richmond, VA 23219
Washington State Arts Commission
711 Capital Way S., Suite 600
PO Box 42675
Olympia, WA 98504-2675
West Virginia Division of Culture
and History
1900 Kanawha Boulevard East
Charleston, WV 25305

13

212-691-6590

614-728-4455

717-705-5644
615-532-5934

512-936-6573

804-225-3132

360-586-2418

304-558-0240