Attachment A - Tables And Figures

ATTACHMENT A:
TABLES AND FIGURES

Mathematica Policy Research
Table B2.1. Overview of APEC- II Study Design
Research Questions/Key
Outcomes

Samples (Completed Sample
Sizes)

Data Collection

Analysis Methods

Objective 1: Generate National Estimates of Erroneous Payments Due to Certification Error
(1) Estimate Erroneous
Payments from
Certification Error
–

–

Amount of
overpayments,
underpayments, sum of
gross overpayments
and underpayments
Erroneous-payment
rate

(2) Estimate Certification
Error by Source of Error
–

Total certification error
rate
– Administrative error
rate
– Household reporting
error rate
(3) Estimate Certification
Error Rate for Directly
Certified Students and
How It Relates to
Implementation Method

Nationally representative
cross-sectional sample of
certified students (n=3,835)
and denied applicants
(n=585) drawn from 130
SFAs and 390 schools

In-person
household surveys
Record abstractions
–
–
–

Application data
Participation data
Change in
certification and
enrollment data

Descriptive tabular
analysis. Separate
estimates for NSLP
and SBP
90% confidence
interval of ±2.5%
around the estimate
of the percentage of
erroneous payments
Conduct comparisons
with APEC

Nationally representative
cross-sectional sample of
certified students (n=3,835)
and denied applicants
(n=585) drawn from 130
SFAs and 390 schools

Record abstractions
(see above)

Nationally representative
sample of directly certified
students (subsample of
certified-free students)

Record abstractions
In-person
household surveys

Combine estimates of
error due to
certification and noncertification error
Descriptive tabular
analysis
Separate estimates
for NSLP and SBP
Conduct comparisons
with APEC
Descriptive tabular
analysis
Separate estimates
for NSLP and SBP

Objective 2: Generate National Estimates of Erroneous Payments Due to Non- Certification Error
Estimate Erroneous
Payments from Noncertification Error
–
–
–

Dollar amounts and
error payment rates
Gross and net error
Separately for cashier
error and three types of
aggregations error;
total error

Nationally representative
sample of school districts
(n=130) and schools
(n=390)

–
–
–
–

1

Observe cashier
transactions
Verify point-ofsale meal counts
Review of meal
count records
Review of
reimbursement
claims

Descriptive tabular
analysis
Separate estimates
for NSLP and SBP
Conduct comparisons
with APEC

Mathematica Policy Research

Research Questions/Key
Outcomes

Samples (Completed Sample
Sizes)

Data Collection

Analysis Methods

Objective 3: Refine Existing Estimation Models for Updating Annual Estimates of Erroneous
Payments Based on Extant Data
Annual Estimates of
Erroneous Payments
(certification and noncertification error)
Amount of overpayments
Amount of underpayments

Nationally representative
sample of students certified
for free and reduced-price
meals and denied applicants
from study districts
District-level data on all
districts in United States

Gross total sum of
overpayments and
underpayments

Data collected from
school districts and
households in SY
2012-13
Extant data on
districts (from FNS742 and other
sources)

Regression modeling
and estimation
Separate models for
NSLP and SBP
Model validation

Objective 4: Explore Alternatives for Generating State- Level Estimates of Erroneous Payments
Create and Validate
Models for Generating
Annual Estimates of
Erroneous Payments at the
State Level

Not applicable

Extant data on
districts (from FNS742 and other
sources)

Regression modeling
and estimation
Separate models for
NSLP and SBP
Model validation

Produce Separate Estimates of Erroneous Payments for LEAs Participating in the Community
Eligibility Option
Estimate Erroneous
Payments Due to
Certification and Noncertification Error

45 SFAs; 135 schools; 2,160
students (directly certified,
certified by application, paid
students)

State/local SNAP
participation lists

Descriptive tabular
analysis

Collect noncertification error
data

Separate estimates
NSLP and SBP

APEC = Access, Participation, Eligibility, and Certification study of 2007
LEAs = local education agencies
NSLP = national school lunch program
SBP = school breakfast program
SFA = school food authorities
SNAP = supplemental nutrition assistance program.

2

Mathematica Policy Research
Figure B2.1. Summary of APEC- II Study Samples (National and CEO Completed Sample Sizes)

APEC –II Study Sample
n = 166 SFAs
SFA Survey
National
Noncertification
Error
Cashier error
Aggregation errors

Onsite Sample
n = 130 SFAs
n = 390 schools
Cashier observations
Record abstraction

Certified
Free/Reduce Price
Students
n = 3,835
In-home survey
Record abstraction

Denied Application
n = 585
In-home survey
Record abstraction

CEO Sample
n = 45 SFAs
n = 135 schools
n = 2,160 students

National
Certification
Household reporting
error
Administrative error

Certification
Noncertification
Error in CEO*

*Data collection related to noncertification error will be limited to a subset of 15 SFAs and 45 schools in the CEO
sample.
CEO = Community Eligibility Option; SFA = school food authorities.

3

Mathematica Policy Research
Table B2.2. Overview of Data Collection
Instrument and Team
Conducting the Data
Collection
School Food Service
Authority Survey (SFA
Director Survey)
(Westat)

Respondent/Data
Source (Mode)

SFA director (selfadministered
questionnaire)

Sample, Response
Rate, Expected
Completes

Main sample: 130
CEO sample: 45

Response rate: 95%
Completes: 166

Student Sampling—
non-CEO Schools
(Mathematica)

SFA director
(in person)

n.a.

Student Sampling—
CEO Schools
(Mathematica)

LEA/SFA director
(Telephone)

n.a.

Household Survey
(Mathematica)

Parent/guardian
(in person)

Free/Reduced: Main
sample: 4,794
Denied Applicant:
Main sample: 731
Response rate: 80%
Completes: 4,420

Record Abstraction
(Westat)

No respondent
(obtain copies of
records or in-person
abstraction from
applications and
direct certification
documents onto
abstraction forms)

Free/Reduced: Main
sample: 3,835
Denied Applicant:
Main sample: 585
Response rate: 100%
Completes: 4,420

4

Instrument
Components

For SFA and sampled
schools: district and
school
characteristics,
student enrollment,
certification status
and method certified,
meals served, direct
certification
procedures, and
other relevant
information
Lists of free,
reduced-price, and
denied applicant
students and directly
certified students in
up to 390 schools
Lists of enrolled
students; lists of
free, reduced-price,
and paid students
and directly certified
students in 45
districts and 135
schools
Household size and
composition; student
participation in SBP
and NSLP; income
sources and amounts
received for each
household member
aged 16 or older
(from visual
verification of pay
stubs or other
documentation); and
participation in SNAP,
TANF, and FDPIR;
participation in
homeless, runaway,
or migrant programs
Student’s identifying
information,
household
composition and
income, qualifying
program
participation, and
certification decision
and reason for
approval or denial

Key Data Elements

For SFA and sampled
schools: number of
enrolled students,
number of
applicants, number
of certified students
by certification status
and method certified,
number of meals by
reimbursement type,
characteristics of
certification and
verification
procedures
Student name,
contact information,
certification status,
method certified,
date certified, school
attending
Student name,
certification status,
method certified,
date certified, school
attending

Household income;
family size; NSLP and
SBP participation;
perceptions of meal
programs;
participation in SNAP,
TANF, and FDPIR

Student and
household
identification
information, number
in household, income
of household
members, household
program
participation,
verification
information

Mathematica Policy Research

Instrument and Team
Conducting the Data
Collection

Meal
Claiming/Counting
(a) Records and Other
Reviews
(Westat)

Respondent/Data
Source (Mode)

SFA director; school
food service manager
(in person)

Sample, Response
Rate, Expected
Completes

Records for a target
day, week, or month,
as appropriate
Main Sample: 390
CEO sample: 45

Instrument
Components

Meal counts by
reimbursable meal
category for sampled
schools and SFAs by
target day, week, or
month

Errors in transcribing
and totaling data
from individual cash
registers, errors in
reporting meal
counts to SFAs,
errors in reporting
claims to state
agencies for meal
reimbursement

For sampled
“trays/transactions,”
foods and amounts
selected; recipient
(student or
nonstudent); how
cashier classified
meal (reimbursable
or nonreimbursable)

Cashier error:
incorrectly classify
meal as reimbursable
when it is not;
incorrectly classify
meal as
nonreimbursable
when it is

Response rate: 100%

Meal
Claiming/Counting
(b) Cashier
Transactions
Observation
(Westat)

Cashier
(in person)

Completes: 435
Schools
Main Sample: 19,500
lunch transactions;
19,500 breakfast
transactions
CEO Sample: 2,250
lunch transactions;
2,250 breakfast
transactions

Key Data Elements

Response rate: 100%

Extant Data
(Mathematica)

(1) CCD (not
applicable)
(2) Census data (not
applicable)
(3) Administrative
data from FNS and
other agencies (not
applicable)

Completes: same as
samplea
n.a.

Locale, enrollment,
percent certified for
FRP lunch, grade
span of district, Title
I status of schools,
poverty rates, income
levels, verification
results, eligibility
determinations
made, NSLP /SBP
certification rates,
NSLP /SBP
participation rates

a
50 lunch transactions and 50 breakfast transactions per school covering target day during week from each sampled
school.

n.a. = not applicable.

CCD = Common Core of Data; CEO = Community Eligibility Option; FDPIR = Food Distribution Program on Indian
Reservations; FNS = Food and Nutrition Service; LEA = local education agency; NSLP = National School Lunch Program;
SBP = School Breakfast Program; SFA = school food authorities; SNAP = Supplemental Nutrition Assistance Program;
TANF = Temporary Assistance to Needy Families program.

5