Field Test Nonresponse Bias Analysis

MEMORANDUM

955 Massachusetts Avenue, Suite 801
Cambridge, MA 02139
Telephone (617) 491-7900
Fax (617) 491-8044
www.mathematica-mpr.com

TO:

Mark Denably, Economic Research Service

FROM:

Nancy Cole and John Hall, Mathematica Policy Research

SUBJECT:

FoodAPS Field Test – Nonresponse Bias Analysis

10/10/2011
Revised 03/08/2012
DATE:

The field test of the National Household Food Acquisition and Purchase Survey (FoodAPS),
was conducted from February through May 2011. This memo reviews survey response rates and
presents the results of the nonresponse bias analysis.
BACKGROUND
The field test was conducted in two purposively selected PSUs in the mid-Atlantic region.
Within each PSU, 8 Secondary Sampling Units (SSUs) were selected as local survey areas. Within
each sampled SSU, we sampled addresses for screening from a sampling frame constructed from
two sources:
SNAP List. A list of addresses for SNAP recipients (in each SSU), obtained from the
State SNAP agency; these addresses were used in selecting households expected to be
receiving SNAP.
Non-SNAP List. A commercial list of addresses (in each SSU), known as an AddressBased Sampling (ABS) file, compiled from the United States Postal Service Delivery
Sequence File. These addressed were matched against those on the SNAP list, and the
SNAP list addresses were flagged. The flagged addresses were sampled at a different rate
than those appearing only on the ABS file.
All sampled addresses were randomly grouped into replicate subsamples and randomly assigned
to one of two survey protocols (Single Book or Multiple Book) and one of two incentive levels (low
or high). Field interviewers confirmed the presence of an occupied dwelling unit at each sampled
address and administered a screener to determine the household’s eligibility for the survey. Eligibility
was determined by membership in a quota group:
1. Quota group A – Non-SNAP, household income ≤ 100% of poverty
2. Quota group B – Non-SNAP, household income between 100-185% of poverty
3. Quota group C – Non-SNAP, household income

185% of poverty (not eligible)

4. Quota group D – SNAP participant household
Households in Quota group C were not eligible for the field test. Households screened into
quota group B from releases 2 and 3 were also not eligible for the field test.

2
Table 1 shows unweighted and weighted response rates for the addresses released to the field.
Response rates are calculated for each stage of response, overall and by experimental groups and
sampling frame.
Table 1. Response Rates to the FoodAPS Field Test

Weighted Response Rates
Overall
response
rate,
unweighted

Overall

97.1

96.79

96.68

96.90

96.80

96.78

98.16

96.68

83.1

84.93

84.73

85.12

85.15

84.72

89.12

84.57

72.2

70.25

69.11

71.35

67.59

72.73

76.56

69.68*

Screening response rate
(SRR =
DRR*EDR*SCR)

58.2

57.75

56.61

58.85

55.71

59.63

66.97

56.97

Household interview
completion rates (CR)
HH #1
HH #2
HH #3

62.9
53.6
56.3

60.98
53.26
55.19

60.80
53.01
54.62

61.16
53.51
55.75

56.02
47.03
49.00

65.25*
58.61*
60.50*

67.02
57.40
59.79

60.18
52.71
54.57

Household interview
response rates
(RR = SRR*CR)
HH #1
HH #2
HH #3

36.6
31.2
32.7

35.21
30.76
31.87

34.42
30.01
30.92

35.99
31.49
32.81

31.21
26.20
27.30

38.91*
34.95*
36.08*

44.89
38.44
40.04

34.29
30.03
31.09

Response rate

Dwelling unit
determination rate
(DRR)1
Screener contact rate
(screening eligibility
determination rate)
(EDR)2
Screening completion
rate (SCR)3

Survey Protocol
Single
Book

Multiple
Books

Incentive Level

Sampling Frame

Low

SNAP

High

ABS

*

Indicates statistically significant subgroup differences based on t-tests.
The dwelling unit could not be determined for 58 addresses in locked buildings and gated communities.
2
Eligibility was not determined if the case was untouched (N=1) or expired due to maximum attempts (N=200).
3
Completed screeners as a percentage of eligible addresses.
1

Figure 1 provides a graphical depiction of response at each stage of contact. Of the 2,017
sampled addresses, 50 were released in error or not in the sample frame, leaving 1,967 addresses in
the sample. Of the 1,967 remaining addresses, 1,610 were determined to be occupied dwelling units
and 299 were found unoccupied for a dwelling determination rate of 97.1 percent
((1610+299)/1967). Five stages of response followed the determination of an occupied unit:
screener contact, screener cooperation, agreement to participate in the study, completion of the first
household interview to start the data collection week, and completion of the data collection week. At

3
each stage after the first, we can examine the marginal response (at that stage only) or total response
(cumulative through that stage).
Figure 1. Response by Stage of Contact

Response stage /
N for response analysis
1. Dwelling unit
determination

2. Screener contact
N=1534

Respondents

Nonrespondents

Other

Occupied Dwelling Units

Not determined

Not occupied

(gated/locked buildings)

N=1610

N=58

N=299

Screener contact

No screener contact

Not eligibleb

(effort ended)

N=1333

N=201

Screener complete

Screener refusal

N=962

N=371

Eligible & Agreed

Eligible & refused

N=633

N=99

N=230

5. Start data collection
week
N=556

Completed HH1

Refused HH1

Effort ended

N=461

N=95

N=77

6. Complete data collection
week
N=461

Completed HH3

Did not complete week

N=411

N=50

3. Screener cooperation
N=1333

4. Agree to participate
N=732

N=76

Not eligible
(quota group closed)

b

Cases not eligible for screening include those with language other than English or Spanish, not available during
the survey period, or with physical impairments making participation impossible. For the purpose of subsequent
analysis, we coded those not eligible for screening as having been contacted.

Nonresponse bias analysis includes comparisons of respondents and nonrespondents at each
stage of response. Information available for this comparison varies by the stage of response, as
shown in Table 2. At the first two stages (screener contact and screener cooperation) a comparison
of respondents and nonrespondents is limited to information about the sample frame and the timing
of contacts. We focus on marginal response rates because after the first two stages, we have more
information available for examining marginal response rates (as opposed to total response).
Response at the third and fourth stages (agreement to participate; start of data collection week)
can be assessed using information from the sample frame and the screener. Response at the final

4
stage (completing the study week, among those that start the week) can be assessed using sample
frame information and household characteristics from the first household interview.
Table 2. Information for Assessing Nonreponse Bias

Information available for response bias analysis
Response stage

Data source

Data elements

1. Screener contact

Sample information

PSU, SSU, interviewing team, incentive level

2. Screener cooperation

Sample information

PSU, SSU, interviewing team, incentive level

3. Agreement to participate
at the time of screening
(among eligible
households)

Sample & screener

Sample info plus, language, household size,
SNAP participation, income group, received
study letter, store type for most food shopping,
other types of food stores in last 30 days, food
bank in last 30 days, number of household
members by age group a

4. Start data collection week
(among households that
agree at screening)

Sample & screener

Sample info plus, survey protocolb, language,
household size, SNAP participation, income
group, received study letter, respondent is meal
planner or food shopper

5. Complete data collection
week (among households
that start the week

Sample, screener, and
household interview

Same as above plus, respondent characteristics
(age, gender, education, race, employment
status, and marital status), number of
household members by age group.c
a Information about food shopping, food banks, and household members by age group was obtained from
about half of respondents who did not agree to participate at the time of screening (the other half declined to
answer these additional questions. These items were not collected from respondents who agreed to
participate at the time of screening and later declined before the first household interview.
Survey protocol is known at the time of contact, however, it does not become known to respondents
until after they agree to participate.
b

c Characteristics

are available for all household members, however, degrees of freedom are limited for the
analysis at stage 5.

The following methods are recommended for analysis of nonresponse bias1:
1. Compare the distributions of respondents and nonrespondents across subgroups using
sample frame characteristics
2. Use multivariate analysis to identify characteristics of cases associated with nonresponse

1

Items 1-4 are recommended by the National Center for Education Statistics, Statistical Standards Program:
http://nces.ed.gov/statprog/2002/std4_4.asp

5
3. Compare respondents to known population characteristics from external sources (we do
not use this method for the field test because population characteristics are measured
imprecisely at the block group level2)
4. Compare the characteristics of easy/early completed cases with the characteristics of
difficult/later completed cases (this assumes that nonrespondents are similar to “hard to
reach” respondents)
Distributions of Respondents and Nonrespondents
We used the first method to examine the weighted distribution of respondents and
nonrespondents by frame characteristics and the timing of contacts (Table 3). Chi-square tests are
used to identify statistically significant differences in response by subgroups.
The results in Table 3 show that geography is associated with differences in response rates:
PSU – Differences in response by PSU are not statistically significant at the first two
stages of contact; differences are significant for agreement to participate and starting
a data collection week, with Atlantic county having lower response.
SSU –At every stage of contact, there were statistically significant differences in
response by SSU. SSU and interviewing team are correlated so that it is difficult to
draw conclusions from univariate analyses. One pattern is that SSUs/interviewing
teams with the highest screener contact rates have the lowest screener cooperation
rates, and vice versa.
For the most part, characteristics of the sampling frame other than geography are not associated
with statistically significant differences in response. The incentive level has a statistically significant
impact only at the final stage of response (completing the data collection), although there are large
differences in response for incentive groups at other stages. The SNAP frame has the higher
response rates at every stage of response except the last, though not statistically significant.
Differences between SNAP participants and other target survey groups were not statistically
significant, but there are potentially important differences in response between groups. For example,
the lowest income group is 8 percentage points more likely to agree to the study, compared with the
higher income group, but 11 percentages points less likely to complete (from among those that
start).
Timing during the data collection period was important in four respects:
1. The screener contact rate declined throughout the data collection period, as
expected. This is because the hard-to-reach cases remain in the pool for longest.
Table 3. Percent of Households Responding at Each Stage, By Frame Characteristics and Timing

2

For the full-scale survey, we will compare respondent characteristics with national estimates from the American
Community.

6

Screener contact
Yes

No

Screener cooperation Agree to participate Start data collection
(Among contacted) (Among screened)
(Among agreed)
Yes

No

Yes

No

Yes

No

Complete data
collection
(Among started)
Yes
No

Primary Sampling Unit
Atlantic

92.2

7.8

69.1

30.9

78.3

† 21.7

76.2

† 23.8

87.0

13.0

Essex

85.3

14.7

72.3

27.7

91.6

8.4

87.0

13.0

92.8

7.2

A-1

87.6 † 12.4

82.8

† 17.2

82.9

† 17.1

80.6

† 19.4

A-2

93.2

6.8

74.1

25.9

75.0

25.0

90.2

9.8

76.8

A-3

87.0

13.0

70.4

29.6

69.8

30.2

68.9

31.1

100.0

-

A-4

98.6

1.4

80.3

19.7

81.8

18.2

92.2

7.8

81.2

18.8

A-5

100.0

-

72.7

27.3

86.8

13.2

76.1

23.9

93.3

6.7

A-6

96.4

3.6

61.1

38.9

77.4

22.6

73.1

26.9

76.9

23.1

A-7

88.8

11.2

69.7

30.3

82.1

17.9

73.4

26.6

86.7

13.3

A-8

90.6

9.4

67.9

32.1

75.0

25.0

73.6

26.4

97.2

2.8

E-1

72.4

27.6

69.1

30.9

86.4

13.6

93.8

6.2

80.4

19.6

E-2

79.0

21.0

76.2

23.8

97.6

2.4

82.0

18.0

89.0

11.0

E-3

81.4

18.6

75.9

24.1

100.0

-

86.9

13.1

96.4

3.6

E-4

79.4

20.6

64.0

36.0

97.9

2.1

86.5

13.5

87.4

12.6

E-5

90.0

10.0

77.3

22.7

92.9

7.1

93.8

6.2

94.6

5.4

E-6

82.3

17.7

72.8

27.2

96.1

3.9

93.5

6.5

96.8

3.2

E-7

80.6

19.4

70.1

29.9

98.5

1.5

83.5

16.5

89.5

10.5

E-8

93.7

6.3

74.0

26.0

81.1

18.9

84.1

15.9

98.1

1.9

Team#1

95.4

† 4.6

66.5

33.5

82.1

† 17.9

90.8

† 9.2

82.6

17.4

Team#2

92.2

7.8

68.2

31.8

77.2

22.8

73.1

26.9

87.9

12.1

Team#3

80.0

20.0

78.9

21.1

96.4

3.6

90.6

9.4

92.4

7.6

Team#4

87.0

13.0

71.0

29.0

88.2

11.8

83.3

16.7

93.7

6.3

Low

88.8

11.2

68.6

31.4

82.3

17.7

80.1

19.9

87.5 † 12.5

High

88.1

11.9

72.9

27.1

88.2

11.8

83.9

16.1

92.6

ABS

87.9

12.1

69.7

30.3

83.8

16.2

81.8

18.2

90.8

9.2

SNAP

91.2

8.8

76.6

23.4

92.2

7.8

84.0

16.0

89.2

10.8

-

-

-

-

91.6

8.4

85.3

14.7

91.4

8.6

-

-

-

-

88.3

11.7

78.7

21.3

81.9

18.1

-

-

-

-

80.0

20.0

81.5

18.5

93.5

6.5

Week #1-4

100.0

-

77.5

22.5

82.0

18.0

90.7

9.3

90.4

9.6

Week #5-8

99.7

† 0.3

67.3

32.7

88.2

11.8

68.4

31.6

91.4

8.6

Week #9-12

90.6

9.4

64.8

35.2

81.6

18.4

82.6

17.4

88.0

12.0

Week #13-16

61.6

38.4

80.9

19.1

93.4

6.6

92.0

8.0

93.2

6.8
14.8

Secondary Sampling Unit
78.0 † 22.0
23.2

Field team

Incentive level
7.4

Sampling Frame

Target Survey Groups
SNAP households
Income < 100% FPL
Income 100-185
Data Collection Period

Weekday of final screener status
Sunday

91.3

† 8.7

73.5

26.5

83.5

16.5

89.0

† 11.0

85.2

Monday

89.7

10.3

76.4

23.6

85.2

14.8

86.7

13.3

91.2

8.8

Tuesday

90.0

10.0

69.5

30.5

91.3

8.7

73.6

26.4

85.3

14.7

Wednesday

83.9

16.1

70.4

29.6

85.6

14.4

82.3

17.7

93.7

6.3

Thursday

82.1

17.9

66.2

33.8

83.0

17.0

69.3

30.7

96.9

3.1

Friday

88.0

12.0

73.2

26.8

88.2

11.8

86.8

13.2

94.8

5.2

Saturday

95.3

4.7

68.4

31.6

80.6

19.4

91.0

9.0

85.5

14.5

Note: Significant differences in distributions are noted by †. Differences are tested using chi-square tests.

7
2. All response rates from screener cooperation to completion rose somewhat at the
end of the data collection period (not statistically significant), possibly due to
reductions in field staff, with only the most productive interviewers remaining.
3. Day of the week was significantly related to screener contact rates, with the highest
contact rates on the weekend.
4. The day of the week that the screener was conducted was significantly related to the
likelihood of starting a data collection week. Screeners conducted on Wednesday
and Thursdays were least likely to result in data collection; screeners conducted on
the weekend were most likely to result in data collection.
For the most part, however, this first method of analyzing nonresponse provides little
information about potential bias because sample frame characteristics are limited to geography and
timing.
Multivariate Analysis to Identify Characteristics of Cases Associated with Nonresponse
Multivariate analysis of response was implemented using unweighted logistic regression. At each
stage of response, we modeled the likelihood of response as a function of the characteristics
available for both respondents and nonrespondents (covariates). We examined marginal rates of
response at each stage to make use of additional information available for both respondents and
nonrespondents at each stage.
The tables presented in this section include information for (a) odds ratios, (b) statistical tests of
individual predictors (Wald chi-square tests), (c) overall model evaluation (Likelihood ratio, Score,
and Wald tests), and (d) goodness-of-fit statistics (Hosmer-Lemeshow chi-square test). Statistically
significant predictors are denoted by asterisks on the odds ratio for that predictor.3 For the first two
stages of response, covariates are limited to the characteristics included in Table 3. For stages 3
through 5 (agreement to participate, starting the week, and completing the week), additional
covariates are taken from the screener and the first household interview. All of the tables presented
in this section include multiple specifications to show the sensitivity of results to alternative sets of
included variables. For the most part, we focus attention on statistically significant results, however,
we also observe odds ratios with large magnitude that persist across model specifications and are not
statistically significant. This is likely due to small sample size and limited power.
Table 4 presents the results of logistic regressions for the likelihood of response at every stage,
as a function of sample frame characteristics and timing of contact. We present each model with and

3

Good model fit is indicated by Likelihood ratio, Score, and Wald tests with p-values < .05; individual predictors
are identified in the table as statistically significant if the p-value for the Wald chi-square statistic is <.05; the HosmerLemeshow chi-square test is insignificant at p-value>.05 indicating that we cannot reject the null hypothesis that the
model fit to the data well.

8

Table 4. Logistic Regression Analysis of Response at Each Stage of Contact (Odds Ratios in Table)
Screener contact
Covariate

#1

#2

Screener cooperation

Agree to
participate

Start data
collection

Complete
data
collection

#1

#2

SSU = A-1

1.99

2.02

0.72

0.34

0.39

SSU = A-2

1.22

1.24

0.54

0.56

0.37

SSU = A-3

1.44

1.51

0.39

0.60

1.08E6

SSU = A-4

1.91

1.98

0.80

0.75

0.66

SSU = A-5

1.39

1.43

1.07

0.25

2.56

SSU = A-6

0.83

0.87

0.83

0.34

1.05

SSU = A-7

1.16

1.23

0.81

0.61

1.26

SSU = A-8

1.26

1.33

0.48

0.93

6.83

SSU = E-1

0.33*

0.35*

0.15*

2.16

0.09

SSU = E-2

0.53

0.55

1.60

0.90

0.52

SSU = E-3

0.88

0.89

1.06E6

1.23

1.59

SSU = E-4

0.50*

0.49*

5.39

1.06

0.41

SSU = E-5

0.55

0.57

0.58

1.37

0.76

SSU = E-6

0.40*

0.43*

0.72

1.53

1.93

SSU = E-7

0.68

0.70

9.36*

0.78

0.58

Sample frame characteristics
Atlantic County

1.55

1.07

Team#1

2.65*

0.83

0.61

0.59

1.34

2.62

0.44

Team#2

1.37

1.10

0.64

0.61

1.13

0.81

0.24

Team#3

0.77

0.95

2.89*

2.88*

8.88*

0.96

1.51

High incentive level

0.96

0.95

1.00

1.01

1.54

1.23

1.98*

1.16

0.85

1.76

0.65

0.58

SNAP household

1.33

1.57

0.99

Very low income HH

1.36

1.10

0.44

Multiple Book
SNAP frame

1.68*

0.81

1.27

1.20

Survey Strata

Timing of screener contact
Week #5-8

0.63*

0.64*

2.32*

0.23*

1.31

Week #9-12

0.56*

0.62*

1.57

0.42*

1.43

Week #13-16

0.91

1.09

1.57

1.16

5.73*

Sunday

0.53

0.31*

1.33

1.32

1.08

0.95

1.06

Monday

0.60

0.67

1.55

1.54

1.22

0.92

1.62

Tuesday

0.54

0.50

1.27

1.26

2.30

0.40

1.18

Wednesday

0.22*

0.32*

1.28

1.29

1.60

0.32*

2.96

Thursday

0.21*

0.52

0.92

0.94

1.29

0.29*

3.26

Friday

0.43*

0.59

1.10

1.12

1.88

0.45

4.41

0.98

1.01

1.04

0.93

# Contact attempts

0.78*

Goodness of fit tests
Likelihood Ratio test

95.25*

565.91*

67.06*

68.48*

92.81*

68.58*

56.39*

Score test

92.58*

654.60*

64.58*

65.75*

75.71*

66.97*

53.06*

Wald test

84.03*

287.36*

61.17*

62.23*

52.40*

56.25*

39.02

H-L test
18.77*
10.51
3.94
11.06
10.25
15.04
7.03
Note: Logistic regressions estimate the likelihood of response. All covariates are zero-one dummies except the number
of contact attempts. Table shows odds ratios. Asterisks denote odds ratios from estimates with p-value <.05.
Number of observations included in regressions are N=1610 (screener contact), N=1334 (screener cooperation), N=732
(agree to participate), N=556 (start data collection), and N=461 (complete data collection). Each column after the first
includes analysis of the conditional rate of response, among those responding at the prior stage.

9
without the “number of attempts at contact” (model 1 does not include number of attempts; model
2 includes number of attempts) since this measure is likely to be correlated with observed and
unobserved household characteristics. Table 3 indicates that geography and timing during the field
period affect screener contact. However, because of cells with perfect response (SSU A-5 and Weeks
#1-4) we include only a PSU indicator and do not control for timing in the models of screener
contact. Day of the week is important at the first stage, with Saturday (the left-out group) providing
the best opportunity for contact. Households screened on Wednesday and Thursday are the least
likely to start data collection. After controlling for other factors, the higher incentive has a
statistically significant impact only on the final stage of response (probability of completing the data
collection week).
Tables 5-8 present alternative specifications of logistic regressions for the marginal response at
stages 3-5 (agree to the study, start the data collection week, complete the data collection). Since
SSUs were not significantly related to response in stages 3-5 in Table 4, SSUs are not included in
subsequent analyses.4 In all tables, Model 1 replicates Table 4 without SSUs, and subsequent models
include increasing numbers of covariates.
Table 5 presents logistic regressions for “agree to participate.” Models 2-6 provide consistent
evidence that response was lower in Atlantic county and higher among (a) the higher incentive
group, (b) households reporting receipt of the study letter, and (c) Spanish language respondents.
Study letters were mailed to all sampled addresses and respondents were asked, during the screener,
if they received the letter. Receipt of the letter may proxy for household motivation to participate,
however, we are unable to control for other characteristics to test this hypothesis at this stage of
response. Spanish language respondents may also proxy for the strength of bilingual interviewers
who had more experience on average.
Table 6 presents logistic regressions for “agree to participate and starting the data collection
week” versus “agreeing and not starting the data collection week.” At this stage response was lower
in Atlantic county and among single-person households; higher among the higher incentive group,
those that received the letter, and Spanish language households; and varied by interviewing team.
Table 7 presents logistic regressions for “start the data collection week” among all those who
agree to participate. At this stage of response none of the sample frame characteristics are
significant. Timing of the screener had an impact on the probability of starting the data collection
week: response was lowest in weeks 5-8 (possibly due to the break in activities to conduct the 50case analysis); response was also lowest for screeners conducted in the middle of the week (relative
to Saturday). None of the available household characteristics were significantly related to the
likelihood of beginning the data collection week, among those that agreed to participate.

4

Interviewing team is correlated with geography. Two teams were assigned to each PSU (team 1 and 2 to Atlantic;
Team 3 and 4 to Essex). Teams 1 and 4 worked across PSUs but mostly in their own PSU. Each team in a PSU covered
all SSUs in the PSU but there was some concentration of effort leading to correlation of teams and SSU, especially as we
move across the stages of response to smaller samples.

10
Table 5. Logistic Regression Analysis – Probability of Agreeing to Participate at Screening Among
those Screened and Eligible (Odds Ratios in Table)
Covariate

Model 1

Model 2

Model 3

Model 4

Model 5

Model 6

Sample frame characteristics
Atlantic County

0.37

0.21*

0.19*

0.19*

0.17*

0.17*

Team#1

1.16

1.94

2.06

2.06

2.19

2.21

Team#2

0.88

1.38

1.54

1.52

1.73

1.73

Team#3

2.11

2.53

2.47

2.44

2.53

2.52

High incentive level

1.52

1.74*

1.68*

1.70*

1.71*

1.74*

SNAP frame

1.71

1.90

1.92

1.85

1.83

1.77

SNAP household

1.45

1.30

1.30

1.38

1.09

1.14

Very low income HH

1.59

1.28

1.20

1.20

1.14

1.13

Week #5-8

2.03*

2.32*

1.98

1.94

1.99

1.96

Week #9-12

1.18

1.21

1.03

1.01

1.09

1.06

Week #13-16

1.38

1.18

1.08

1.06

0.97

0.95

Sunday

1.11

0.81

0.89

0.84

0.91

0.85

Monday

1.24

1.17

1.08

1.00

1.11

1.03

Tuesday

2.36

2.05

2.09

1.94

2.12

1.97

Wednesday

1.65

1.60

1.56

1.44

1.61

1.48

Thursday

1.25

1.16

1.08

1.01

1.04

0.96

Friday

1.72

1.87

1.73

1.61

1.82

1.70

# Contact attempts

1.01

1.02

1.02

1.02

1.03

1.03

Received advance study letter

2.19*

2.10*

2.07*

2.20*

2.17*

Addtl units at this address

1.16

1.32

1.48

1.31

1.50

13.01*

13.75*

14.21*

16.66*

16.97*

HH size = 1

0.63

0.64

0.55

0.57

HH size = 2

1.56

1.56

1.36

1.36

HH size = 3

1.82

1.79

1.80

1.80

Survey Strata

Timing of screener contact

Screener responses

Language, Spanish

# Dinners prepared per week

0.99

0.99

Food shopping more than
once a wk

0.62

0.57

Food shopping weekly

0.48

0.45

Food shopping bi-weekly

1.28

1.19

Goodness of fit tests
Likelihood Ratio test

62.41*

97.69*

107.84*

105.88*

116.88*

115.02*

Score test

58.08*

85.60*

96.70*

94.98*

104.23*

102.59*

Wald test

49.81*

69.84*

77.16*

76.05*

82.55*

81.54*

Hosmer-Lemeshow test
10.75
6.11
17.60*
19.21*
5.91
5.57
Note: Logistic regressions estimate the likelihood of response. All covariates are zero-one dummies except where
indicated by “#.” Table shows odds ratios. Asterisks denote odds ratios from estimates with p-value <.05.
Number of observations included in regressions is N=732.

11
Table 6. Logistic Regression Analysis – Probability of Agreeing and Starting Data Collection (HH1
complete) versus Not Agreeing (Refusal Short Form complete) (Odds Ratios in Table)
Covariate

Model 1

Model 2

Model 3

Model 4

Model 5

Model 6

Sample frame characteristics
Atlantic County

0.05*

Team#1

8.10*

0.60

0.15*
0.61

0.56

0.68

Team#2

5.01*

0.35*

0.33*

0.27*

0.38*

Team#3

3.77*

5.16*

4.78*

4.26*

4.71*

High incentive level

2.36*

2.36*

2.31*

2.42*

2.38*

2.28*

SNAP frame

2.19

1.73

1.73

1.63

1.71

1.60

SNAP household

1.88

1.72

2.18*

1.77

2.08

2.01

Very low income HH

1.15

1.15

1.25

1.12

1.18

1.64

Week #5-8

2.11

1.98

1.67

1.96

1.92

1.70

Week #9-12

0.89

0.93

0.77

0.95

0.95

1.06

Week #13-16

1.19

1.26

1.02

1.09

1.31

1.48

Sunday

0.95

1.11

0.91

0.97

0.93

1.60

Monday

1.29

1.33

1.31

1.22

1.35

1.68

Tuesday

1.58

1.56

1.50

1.37

1.30

2.43

Wednesday

1.90

1.65

1.70

1.54

1.59

1.98

Thursday

1.16

1.19

1.16

1.03

1.01

1.42

Friday

1.82

1.94

1.93

1.74

1.65

2.01

# Contact attempts

1.02

1.02

1.02

1.02

1.02

1.03

Received advance study letter

2.48*

2.60*

2.28*

2.77*

2.85*

2.77*

HH size = 1

0.54

0.54*

0.53*

0.51*

0.58

20.40*

24.00*

30.54*

Survey Strata

Timing of screener contact

Screener responses

Language, Spanish

26.99*

4.23E6

# Dinners prepared per week

0.95

0.94

Food shopping more than
once a wk

0.75

0.80

Food shopping weekly

0.53

0.56

Food shopping bi-weekly

1.51

1.69

Short form / HH1 responses
Usual store is supermarket

1.73

1.32

Food bank was visited in past
30 dys

1.08

1.10

Any children

1.45

Goodness of fit tests
Likelihood Ratio test

121.34*

108.36*

110.33*

115.83*

127.60*

86.54*

Score test

102.33*

91.95*

96.87*

98.53*

103.89*

75.86*

73.28*

67.83*

72.24*

72.11*

69.43*

58.62*

Wald test

Hosmer-Lemeshow test
12.76
12.02
9.21
9.20
3.88
8.90
Note: Logistic regressions estimate the likelihood of response. All covariates are zero-one dummies except where
indicated by “#.” Table shows odds ratios. Asterisks denote odds ratios from estimates with p-value <.05.
Number of observations included in regressions is N=556.

12
Table 7. Logistic Regression Analysis – Probability of Starting Data Collection, Among those
Agreeing to Participate (Odds Ratios in Table)
Covariate

Model 3

Model 4

Model 5

Model 6

Atlantic County

0.33

0.33

0.32

0.32

Team#1

3.99

4.02

4.01

4.03

Team#2

1.34

1.35

1.35

1.30

Team#3

1.23

1.23

1.20

1.18

High incentive level

1.26

1.26

1.27

1.27

Multiple Book protocol

1.15

1.15

1.14

1.17

SNAP frame

0.69

0.70

0.68

0.70

SNAP household

1.62

1.62

1.58

1.64

Very low income HH

1.02

1.02

1.01

1.01

Week #5-8

0.26*

0.27*

0.27*

0.27*

Week #9-12

0.46

0.46

0.45*

0.45*

Week #13-16

1.03

1.03

1.04

1.03

Sunday

0.78

0.77

0.73

0.73

Monday

0.86

0.85

0.84

0.83

Tuesday

0.37*

0.37*

0.36*

0.37*

Wednesday

0.31*

0.31*

0.31*

0.30*

Thursday

0.28*

0.28*

0.28*

0.28*

Friday

0.48

0.48

0.48

0.48

# Contact attempts

1.04

1.04

1.04

1.04

HH size = 1

1.10

1.10

1.06

1.12

Language, Spanish

1.17

1.20

1.18

1.17

Sample frame characteristics

Survey Strata

Timing of screener contact

Screener responses

# Dinners prepared per week

0.99

0.99

Food shopping more than once a
wk

1.39

Food shopping weekly

1.05

Food shopping bi-weekly

1.03

Screener respondent = meal
planner or food shopper

2.07

2.00

Goodness of fit tests
Likelihood Ratio test

61.03*

61.04*

62.34*

63.45*

Score test

59.98*

59.93*

61.33*

61.84*

Wald test

52.48*

52.44*

53.53*

53.99*

Hosmer-Lemeshow test
8.00
6.21
6.06
5.03
Note: Logistic regressions estimate the likelihood of response. All covariates are zero-one dummies except where
indicated by “#.” Table shows odds ratios. Asterisks denote odds ratios from estimates with p-value <.05.
Number of observations included in regressions is N=556.

Table 8 presents logistic regressions for “completing the data collection week” among those that
start the week. At this stage of response, geography and interviewing team were not significant and
they were dropped from the models. The high incentive resulted in a near doubling of response at
this stage, and the impact of the incentive is robust to alternative model specifications. Timing of the
screener (and thus start of data collection) has a significant impact, with greater completion rates for

13
households that start data collection close to the weekend. Household size reduces the probability of
completion. The impact of Spanish language at this stage is negative (less likely to complete), in
contrast to the positive impact on response at earlier stages, however it is not robust to other
included variables, casting doubt on the role of Spanish language at other response stages (where
variables for inclusion in the model are not available). The impact of Spanish language disappears
when we control for age and education, with response lower among the elderly and less educated
(age greater than 60 is not statistically significant, nor were other education cutoffs).

Table 8. Logistic Regression Analysis – Probability of Completing Data Collection, Among those
Starting Data Collection (Odds Ratios in Table)
Covariate

Model 1

Model 2

Model 3

Model 4

Model 5

Model 6

Sample frame characteristics
High incentive level

1.97*

1.94*

1.94*

1.91*

1.99*

1.98*

SNAP frame

0.93

0.96

0.96

0.99

1.01

1.05

SNAP household

0.97

0.94

0.83

0.79

0.89

0.86

Very low income HH

0.52

0.51

0.55

0.53

0.64

0.62

Sunday

0.83

0.87

1.09

1.15

1.39

1.45

Monday

1.38

1.40

1.85

1.91

1.96

2.01

Tuesday

1.24

1.27

1.62

1.68

1.61

1.64

Wednesday

2.15

2.22

2.74

2.89

3.17

3.30*

Thursday

3.12

3.20

3.31

3.42

3.81

3.86

Friday

3.42

3.46

4.29*

4.31*

4.46*

4.42*

Survey Strata

Timing of screener contact

Screener responses
Received advance study letter

1.23

1.29

1.23

Household size

0.78*

0.78*

0.37*

0.37*

1.01

1.01

Female

1.77

1.80

1.96

1.99

Age < 30

0.99

0.96

0.92

0.90

Age > 70

0.22*

0.23*

0.12*

0.13*

Race = Black

1.56

1.55

Married

1.30

1.31

Education less than H.S.

0.31*

0.31*

Employed

0.51

0.51

Language, Spanish
Respondent characteristics (from HH1)

Goodness of fit tests
Likelihood Ratio test

18.40

17.99

32.27*

31.71*

52.01*

51.65*

Score test

18.25

17.86

34.72*

34.13*

57.07*

56.76*

Wald test

16.79

16.45

29.48*

29.07*

43.51*

43.54*

Hosmer-Lemeshow test
6.45
9.40
5.40
11.02
3.08
7.18
Note: Logistic regressions estimate the likelihood of response. All covariates are zero-one dummies. Table shows odds
ratios. Asterisks denote odds ratios from estimates with p-value <.05. Number of observations included in regressions
is N=461.

14
Across all stages of contact, the main predictors of response are: geography at the early stages
(Atlantic City households were hard to contact); timing of contact by day of week (for initial contact
and likelihood of starting and completing the week); incentive level (for every stage after screener
cooperation); receipt of the advance letter (this may be a proxy for interest in the study); and Spanish
language (this may proxy for more experienced interviewers or other unmeasured household
characteristics). To the extent that Spanish language proxies for less education, we may not be
worried about response bias (positive) at the point of starting the data collection week because this is
a group that we consider hard to reach. The lower completion rate among the elderly and less
education is a cause for concern (Table 8) and suggests that additional field interviewer follow-up
during the data collection week may be warranted for these groups of respondents.
It is worth noting that the screener questions we used to measure response bias (short form for
respondents who do not agree to participate) are not significantly related to response. However,
those questions were completed by only half of eligible households who refused to participate, and
were not completed by any of the respondents who initially agreed and refused at the time of the
first household interview.
Easy/Early Completed Cases Compared with Difficult/Later Completed Cases
The final method that we used to examine nonresponse bias is a comparison of the
characteristics of easy/early completed cases with the characteristics of difficult/later completed
cases. This method assumes that nonrespondents are similar to “hard to reach” respondents.
We have two options for defining easy/early cases: (a) elapsed days from sample release to
recruitment into the study (or completed screener), or (b) number of attempts at contact prior to
complete screener. Table 9 shows the schedule of sample release to the field. Most sample was
released within the first two weeks of the field period and interviewers were allowed to work cases in
the most efficient manner to conserve travel time. For this reason, elapsed time since sample release
is not a good measure for identifying easy/early cases.5
The number of attempts at contact prior to completing the screener provides a measure of
“easy” cases that is consistent across cases (this measure does not count idle time when the case was
not being worked). For the full-scale study we plan to work all cases through 8 attempts in phase 1.
Cases not contacted in phase 1 will form the sampling frame for phase 2, with a sample of the hardto-reach cases worked through additional attempts. In the field test, 75 percent of completed cases
were contacted with 8 or fewer attempts; 83 percent were contacted with 10 or fewer attempts; and
90 percent were contacted with 13 or fewer attempts.

5

The distribution of completed cases by elapsed time since release was: 30 percent within 16 days of release, 50
percent within 43 days of release, and 75 percent within 64 days of release, 90 percent within 80 days (maximum days in
the field was 102 days).

15
Table 9. Timing of Sample Release
Release

Replicates

Number of
sampled
addresses

Mailing date for
advance letters

Date sent
to field

Comments

1

1-9

1,223

January 14

January 31

12 of 16 SSUs

1

1-9

423

January 23

January 31

3 SSU

1

1-9

151

February 8

February
14

1 SSU

2

10-15

1,104

February 8

February
14

SNAP & ABS frames
released.
On March 13, we pulled
back all non-SNAP cases
leaving 118 SNAP
addresses from release 2.

1

From listing

55

March 23

March 30

3

16-20

102

April 9

April 14

1-18

2,017

Total in
field

SNAP only

Table 10 shows the results of logistic regression with three definitions of the “easy” cases
(dependent variable). Among completed cases, the probability of a successful contact with eight or
fewer attempts is higher for Hispanic households and lower for single person households and
employed respondents. The odds ratios for female, elderly (age > 70), and education less than high
school all indicate that these respondents are easy to contact, but these characteristics are not
statistically significant and dissipate somewhat by 10 attempts. These results suggest that the planned
cutoff for phase 1 should potentially be raised from 8 to 10 attempts; however, the implementation
of two-phase sampling is being done in conjunction with other changes to screening procedures
(requiring a certain number of attempts in different time periods) so that field test results do not
necessarily provide a reliable guide for the phase 1 cutoff.
No household characteristics are statistically significant in predicting “easy” cases defined by 10
or fewer attempts for a successful contact, although the odds ratio for Hispanic is still large. There
were only 22 complete cases with Hispanic households (from among 412 completed cases) and all
Hispanic cases were completed with 13 or fewer attempts, so that covariate is not included in the
final model. The 10 percent of completed cases requiring more than 13 attempts had a
disproportionate percentage of married respondents, as reflected in the final column of Table 10.

16
Table 10. Logistic Regression Analysis – Probability of Completing With Few Attempts (Odds Ratios
in Table)
Covariate

Easy ≤ 8 attempts

Easy ≤ 10 attempts

Easy ≤ 13 attempts

Sample frame characteristics
Team#1

2.27

6.19*

6.70

Team#2

0.78

0.57

0.94

Team#3

0.57

0.58

0.57

High incentive level

0.96

1.24

1.40

SNAP frame

1.62

1.43

1.81

SNAP household

0.69

1.06

0.76

Very low income HH

2.04

2.02

1.74

Received advance study letter

1.07

1.32

1.15

HH size = 1

0.36*

0.43

0.36

HH size = 2

0.91

1.49

1.40

HH size = 3

1.30

1.18

1.14

Language, Spanish

0.61

0.66

0.54

Age < 30

0.79

0.98

0.61

Female

1.76

1.29

1.22

Age > 70

3.15

1.92

1.10

Hispanic

8.60*

7.10

Race = Black

0.65

0.60

Married

0.88

0.79

0.40*

Education less than H.S.

1.43

1.85

1.57

Employed

0.59*

0.88

0.59

Survey Strata

Screener responses

Respondent characteristics (from HH1)

Goodness of fit tests
Likelihood Ratio test

50.73*

40.27*

29.69*

Score test

45.70*

34.29*

27.27

Wald test

39.50*

28.68

22.90

Hosmer-Lemeshow test

15.12

6.40

10.31

Note: Logistic regressions estimate the likelihood of completing data collection with fewer attempts (as defined in
column headers) All covariates are zero-one dummies. Table shows odds ratios. Asterisks denote odds ratios from
estimates with p-value <.05. Number of observations included in regressions is N=411.

Impact of Higher Incentive on Nonresponse Bias
The field test included an experimental test of the impact of incentive on response. Evaluation
of the impact of the higher incentive level on nonresponse bias is limited because: (1) the incentive
level for the main data collection was not known to the respondent before they completed the
screener, thus the incentive level could only affect response among those who completed the
screener and were eligible for the main data collection; (2) household and respondent characteristics

17
that are useful for non-response bias analysis were not measured until the first household interview,
after the respondent agreed to participate in the study. As discussed above, the incentive level has a
statistically significant impact at multiple stages of response. Tables 3 and 4 indicate that the higher
incentive significantly affects response only at the last stage(completing the data collection). In all
tables after Table 4, we control for household characteristics and find that the higher incentive
significantly impacts the probability of agreeing to participate (Table 5), agreeing to participate and
starting the data collection week (Table 6), and completing the data collection week (Table 8).
Although it is not shown in the tables, we note that the increased response for the higher incentive
level was consistent across sites and across subgroups noted in the tables (language, whether black
and whether education was more or less than high school). This section examines the impact of the
higher incentive on nonresponse bias. As recommended by OMB, we use the following methods:
1. Multivariate modeling of response using respondent and nonrespondent frame variables
to determine if nonresponse bias exists and varies by incentive level (Guideline 3.2.9).
2. Examination of item nonresponse to determine if it is random, or varies by incentive
level (Guideline 3.2.10).
These two methods are used to test the hypothesis that the higher incentive, by bringing more
people into the study through increased response, also moderates response bias.
Impact of Incentives on Nonresponse Bias – Evidence from Frame/Screener Variables
We reran logistic regression models from Tables 5-8 separately for subgroups defined by
incentive level to determine if nonresponse bias varies by incentive level. These results are presented
in Tables 11 and 12. For each stage of response, these tables include three columns with results for
the full sample, low incentive group, and high incentive group.
The first three columns of Table 11 present logistic regressions for “agree to participate.”
Among the full sample, response was lower in Atlantic county and higher among (a) the higher
incentive group, (b) households reporting receipt of the study letter, and (c) Spanish language
respondents. Nonresponse bias associated with receipt of the study letter and Spanish language are
moderated by the higher incentive (Odds ratios on these variables are statistically significant for the
low incentive group, but reduced in magnitude and not significant for the high incentive group.) In
addition, although the odds ratio for additional units at address was not statistically significant for
either incentive group it is substantially smaller for the high incentive group than for the low
incentive group.
The last three columns of Table 11 present logistic regressions for “agree to participate and
start the data collection week” versus “agree and do not starting the data collection week.” Among
the full sample, response was lower in Atlantic county and among single-person households; higher
among the higher incentive group, SNAP households, those that received the letter, and Spanish
language households. Nonresponse bias associated with receipt of the study letter, household size,
and Spanish language are moderated by the higher incentive (these characteristics are not statistically
significant for the high incentive group and the odds ratio is close to one). The greater propensity to
respond among SNAP households is even stronger with the high incentive; however, this is not a
concern because the SNAP group will often be analyzed separately, and when it is combined with
other groups the relative sizes of those groups can be controlled by post-stratifying the weights.

Table 11. Logistic Regression Analysis – Probability of Agreeing to Participate, and Probability of
Agreeing and Starting the Study: Overall and by Incentive Group (Odds Ratios in Table)
Agree to Participate at Screening
(Table 5, Model2)
Covariate

Agree and Start Data Collection
(Table 6, Model 3)

All

Low

High

All

Low

High

Atlantic County

0.21*

0.23

0.20

0.15*

0.11*

0.18*

Team#1

1.94

2.18

1.38

Team#2

1.38

1.74

0.94

Team#3

2.53

8.62*

0.77

High incentive level

1.74*

SNAP frame

1.90

1.92

1.69

1.73

1.19

2.84

SNAP household

1.30

0.89

2.42

2.18*

1.99

3.38*

Very low income HH

1.28

0.86

2.31

1.25

1.08

1.75

Week #5-8

2.32*

1.92

5.10*

1.67

1.05

4.29*

Week #9-12

1.21

0.82

2.24

0.77

0.33

2.05

Week #13-16

1.18

0.87

1.91

1.02

0.66

1.91

Sunday

0.81

0.87

0.52

0.91

0.89

0.61

Monday

1.17

1.25

0.94

1.31

2.03

0.73

Tuesday

2.05

1.76

2.11

1.50

1.60

1.19

Wednesday

1.60

0.75

3.51

1.70

0.76

4.79*

Thursday

1.16

1.02

1.39

1.16

1.00

1.24

Friday

1.87

1.90

1.52

1.93

1.90

1.54

# Contact attempts

1.02

1.05

0.99

1.02

1.03

0.98

Received advance study letter

2.19*

2.85*

1.87

2.28*

3.33*

1.85

Addtl units at address

1.16

3.33

0.64

13.01*

19.67*

8.11

Likelihood Ratio test

97.69*

66.71*

Score test

85.60*

54.03*

Wald test

69.84*

40.48*

Sample frame characteristics

2.31*

Survey Strata

Timing of screener contact

Screener responses

HH size = 1
Language, Spanish

0.53*

0.36*

24.00*

17.60*

0.81

47.28*

110.33*

69.56*

53.83*

43.72*

96.87*

59.59*

47.02*

35.74*

72.24*

42.65*

31.39*

--

Goodness of fit tests

Hosmer-Lemeshow test
6.11
2.18
3.97
9.21
4.52
3.04
Note: Logistic regressions estimate the likelihood of response. All covariates are zero-one dummies. Table shows odds
ratios. Asterisks denote odds ratios from estimates with p-value <.05. Number of observations included in regressions
is N=461.

Table 12 presents logistic regressions for “starting the data collection week” among all those
who agree to participate. At this stage of response none of the sample frame characteristics are
significant, but timing of the screener had an impact on the probability of starting the data collection
week. The last three columns of Table 12 present logistic regression for the probability of
completing data collection. Among the full sample, completing data collection is less likely for (1)
single-person households, (2) less educated respondents, and (3) elderly respondents. The estimates
for single-person and less educated are not significant among the high incentive group. It appears
that nonresponse bias associated with elderly is stronger in the high incentive group. However, the

19
small sample sizes at this stage of response provide unreliable estimates. Of the 461 households
starting data collection, only 19 were elderly and 14 of the 19 completed the data collection week
(74%); 8 of 10 elderly were completes in the low incentive group and 6 of 9 elderly were completes
in the high incentive group.
Impact of Incentives on Adherence to Survey Protocols
A key indicator of respondent adherence to survey protocols is whether respondents save
receipts. Respondents are asked to save receipts for purchases; no receipt is expected for free food
and school meals. A significant percentage of acquisitions were acquisitions for which we do not
expect a receipt.
The high incentive group was more likely to report FAH purchases for which receipts could
be saved and less likely to report FAFH purchases for which receipts could be saved (Table 13).
Examination of adherence to protocols is complicated because the probability of saving a receipt is
conditional on having made purchases for which receipts are available. A proper modeling of this
conditional relationship is beyond the scope of this memo. For our examination of nonresponse, we
use multivariate logistic regression to examine (a) the impact of the incentive on the probability of
reporting FAH and FAFH purchases, and (b) the impact of the incentive on the probability of
saving receipts, among households with any purchases.
The higher incentive was associated with a higher probability of reporting FAH purchase (odds
ratio is 1.76 and statistically significant). The higher incentive did not have a statistically significant
impact on the probability of reporting FAFH purchases, saving FAH (conditional on making a
purchase), or saving FAFH receipts (conditional on making a purchase).6

6

The multivariate logistic regressions controlled for incentive group, SNAP frame, SNAP household, very low
income HH, HH size, Spanish language, Age< 30, Female, Age > 70, Race = Black, Married, Education less than
HS, and employed. Only the following were statistically significant: very low income households were less likely to
report FAH purchase (odds ratio =.28), respondents with education less than HS were less likely to report FAFH
acquisitions (odds ratio=.37); lower education was associated with a lower probability of saving FAH receipts (odds
ratio=.22); SNAP frame was associated with a lower probability of saving FAH receipts (odds ratio=.50).

20
Table 12. Logistic Regression Analysis – Probability of Starting Data Collection (Among those
Agreeing) and Probability of Completing Data Collection: Overall and by Incentive Group
(Odds Ratios in Table)
Probability of Starting (among those
that agree) (Table 7, Model 5)
Covariate

All

Low

High

Atlantic County

0.32

0.83

0.09*

Team#1

4.01

5.26

8.38*

Team#2

1.35

0.37

5.73

Team#3

1.20

1.42

1.05

High incentive level

1.27

Multiple Book protocol

1.14

0.63

1.84

SNAP frame

0.68

0.28*

1.51

SNAP household

1.58

1.87

Very low income HH

1.01

1.20

Week #5-8

0.27*

0.45

0.15*

Week #9-12

0.45*

0.52

0.48

Week #13-16

1.04

6.13

0.38

Sunday

0.73

1.37

Monday

0.84

4.58

Tuesday

0.36*

Wednesday

Probability of Completing Data
Collection (Table 8, Model 5)
All

Low

High

1.01

1.14

0.94

1.10

0.89

1.64

0.36

0.80

0.64

0.70

0.39

0.28

1.39

0.87

2.70

0.19

1.96

1.29

4.28

0.86

0.18*

1.61

1.78

1.35

0.31*

0.56

0.12*

3.17

2.01

7.95*

Thursday

0.28*

0.31

0.18

3.81

8.76

1.60

Friday

0.48

0.69

0.23

4.46*

9.00

3.08

# Contact attempts

1.04

1.05

1.05
1.23

1.70

0.95

Sample frame characteristics

1.99*

Survey Strata

Timing of screener contact

Received advance letter
Screener responses
HH size = 1

1.06

1.90

0.80

0.78*

0.74*

0.80

Language, Spanish

1.18

0.79

1.92

1.01

1.05

1.28

Age < 30

0.92

0.90

0.74

Female

1.96

1.72

2.33

Age > 70

0.12*

0.32

0.02*

Race = Black

1.56

1.57

1.95

Married

1.30

0.82

2.49

Education less than H.S.

0.31*

0.30*

0.30

Employed

0.51

0.72

0.32

Screener respondent = meal
planner or food shopper

2.07

9.72*

1.11

Goodness of fit tests
Likelihood Ratio test

62.34*

59.61*

38.47*

52.01*

32.77*

29.29

Score test

61.33*

52.82*

37.95*

57.07*

31.86*

32.47*

Wald test

53.53*

38.57*

31.46

43.51*

24.34

23.26

Hosmer-Lemeshow test
6.06
5.64
11.47
3.08
7.21
5.77
Note: Logistic regressions estimate the likelihood of response. All covariates are zero-one dummies except where
indicated by “#.” Table shows odds ratios. Asterisks denote odds ratios from estimates with p-value <.05.
Number of observations included in regressions is N=556.

Table 13. Percentage of Households with Saved Receipts
Covariate

Incentive level
Low

High

72.4

80.2

Food at Home (FAH)
Saved any receipts
Yes
No

9.1

7.1

18.5

12.7

Yes

60.8

54.5

No

20.8

22.5

No FAH purchases
Food away from home (FAFH)
Saved any receipts

No FAH purchases
18.4
23.0
Notes: Percentages are weighted. Free acquisitions such as school meals, food from
food banks, and gifts from friends are not expected to have a receipt.

Summary
Key findings from the nonresponse bias analysis are:
1. Screener contact rates varied by survey area and interviewing team. One SSU in each
county provided significantly easier contacts; Saturday was the best day for contacts; there was a
pattern of SSUs/interviewing teams with the highest screener contact rates having the lowest
screener cooperation rates, and vice versa.
This finding points to a need for better management of the screening effort to achieve
consistent effort across interviewing teams. Mathematica is implementing a new web browser
based system for interviewers to use to track their contact attempts. The system will provide
real-time reports for management and feedback to interviewers to help them reach a target
number of contacts per case per time period (2 contacts per case in each of 4 time slots:
weekday morning, weekday evening, weekend day, weekend evening).
2. SNAP frame required fewer contact attempts and had higher screener contact and screener
cooperation rates, with little difference between SNAP and ABS for agreement to participate
and completion rates.
3. Receipt of the study letter was either a critical determinant of response, or a proxy for
household willingness to participate (it is possible that the letter was remembered only by those
interested in the study). The potential importance of the letter indicates a need for revised
procedures.
For the full-scale study, the advance letter will be replaced by a full-color postcard so that
households cannot discard the mailing without opening it. In addition, we will release smaller
batches of sample to the field so that households are contacted soon after receipt of the mailing.

22
4. Lowest income households were least likely to start and complete the data collection week,
however, multivariate analysis (Table 8) indicates that this is largely due to difficulty among the
elderly and less educated.7
This finding points to a need for field support during the data collection week for vulnerable
households (elderly and less educated). During the field test, the primary method of supporting
households during the data collection week was via telephone. There were few instances of field
interviewer visits to households in mid-week as a follow-up to problems reported by telephone.
For the full-scale study, we will revise the protocols that we provide to telephone interviewers
for notifying the field. We will also setup automatic notification to field interviews if the
household does not complete a food reporting call by mid-week, so that field interviewers
follow-up with these households.
5. The higher incentive resulted in higher rates of agreement to participate (Tables 5 and 6) and
higher completion rates (Table 8). In addition, household size was significantly related to
response (Table 6) and completion (Table 8) indicating the value of the incentives for additional
household members.
The higher base incentive was adopted for the full-scale study, and the study will experiment
with two levels of “additional household member” incentives during the first half of the study
period.
6. Timing during the week mattered. Households were asked to begin data collection on the
day following screening. Thus, if screening was done in mid-week, data collection would begin
mid-week. The likelihood that households agreed to participate (Table 5) was somewhat lower if
screened on the weekend (not statistically significant). The probability of starting data collection
was lowest (and significant) if screened on Tues-Thurs, perhaps reflecting the fact that
respondents do not have time in mid-week to review study materials and initiate this effort. The
likelihood of completing the week, once started, was not related to the day that it was started.
During the field test, interviewers were trained to try to begin the data collection right after
screening whenever possible (this involved training the household and conducting the first
interview). Appointments were made to come back and start the process on another day, but for
fear that we would lose the household, this was not encouraged. We will examine the data in
more detail (rates of response and number of acquisition by household that did and did not start
immediately after screening) to determine if a shift in policy is advised so that households can
begin data collection on a day of the week that is most convenient for them. Since all
households provide seven days of data, food acquisitions will represent the full week, regardless
of when households begin the process.

7

Lowest income households (income<100%FPL) were more likely to agree to the study than higher income
households (income 100-185% FPL) and were on par with SNAP households for screener cooperation.

23
7. Higher incentive moderates nonresponse bias. Household characteristics that have a
statistically significant relationship with response, at various stages, are either not significant
among the high incentive group, or significantly moderated by the higher incentive.