Attachment M - CARI Evaluation Report

Attachment M
UNITED STATES DEPARTMENT OF COMMERCE
Economics and Statistics Administration

U.S. Census Bureau
Washington, DC 20233-0001
OFFICE OF THE DIRECTOR

SEP 1 1 2014
MEMORANDUM FOR

Jason Fields
Survey Director, Associate Director for Demographic Programs
Division

From:

Evaluation of the Impact of Computer Audio-Recorded
Interviewing (CARI) on 2013 Survey oflncome and Program
Participation Event History Calendar (SIPP-EHC) Response Rates
and Quality Measures

Subject:

The Evaluation of the Impact of Computer Audio Recorded Interviewing (CARI) on the 2013
Survey oflncome and Program Participation Event History Calendar (SIPP-EHC) Response
Rates and Quality Measures is attached. Questions and comments may be directed by phone or
email to Rachel Bray at (301) 763-2631.

Attachment:
Evaluation of the Impact of CARI on 2013 SIPP-EHC Response Rates and Quality Measures
cc:
J. Farber
T. Mattingly
M. Sundukchi
J. Scott
A. Dajani
P. Singer
S. Thorpe
R. Bray
K. Mathews
K. Staine

(DSMD)

"
"
"
"
"
"
"
"

J. Fields
M. Maday
D. Doyle
L. Bynum
R. Roberts
J. Christy
M. Weiler
A. Walker
D. Morgan

(ADDP)

"
"
"
"
(FLD)

"
"
"

"

USCENSUSBUREAU
Helping You Make Informed Decisions

www.census.gov

of the Imtntct
Income and •»"''"w"""""' Pa.rtidpation
Rates and
Measures
L

(CARI) on the 2013
Histoi-y Calendar

Executive
The use of Computer Audio Recorded Interviewing (CARI) as a tool thr monitoring
interviewers has been shown to be feasible through
Census tests.
unclear.
test
of
CARI on data quality
Wave 3
of respondents selected the 2011 panel of the SIPP-EIIC test and
2 interviews respondents selected for
2012 CARI
test. AH respondents
2013
were asked the consent question, and those that agreed were
in low-income areas
frJr a subset of states.
such, all
SIPP-EHC
measures and comparisons are unweighted and should not
to make inforences
introducing
surveys of
tLS. household population. The 2013
following major findings:
L

2.
their
third wave data collection
rate from 73, percent

3.

31.61 percent to
to CARL

percent,

4.
·'··""V"'''··'"''··'""'u interviewers "-'V''··"·'-''"''"'"'

and a larger

'''"'-'""'<,\.""'~' with

5. There were substantial
fiot1sel1old,
a.:nd. "''"'""'"'

~·.•v•··"-'·'

cooperation.
cooperation rates were
with

1

interviewer
ranging
0.434 to 0.487. 'I'his means
the
alone may explain over forty percent of the variation in CARI cooperation. This may
be
to
in interviewer workloads (Le. some
may have
to
consent
or differences

There are significant difforences in
consent propensity by
· respondent, household, geographic,
characteristics. Persons not
m
U.S.
of Hispanic ethnicity are less likely to
consent to
recorded
persons living outside of cities are more
to
persons living cities. Households in
poorly
or evidence of
likely to provide consent to
recorded;
poor
such as pest problerns or
were more likely to consent be recorded.

has

"'"...,..,,.,,,.,.,,"'"' to assess
traditional SIPP
SIPP-EHC test ~vere

2

capability to
the potential

costs.

event

201 l test consisted of a sample of approximately 4,000 addresses across twenty states,
2012 CARI test consisted of a
of approximately 1,300
across
while
ten states.
The 20 CARI test aHowed for portions of the respondent's interview to be recorded
The CARI
question was asked each time the interviewer
upon their
instrument, and
with a new
to l)e ,.,,,,.,,.....,.,,"'"

was automatically
interview proceeded as nonnal. without
not provide consent
interviews, the respondent
if the respondent
if
respondent had already answered the consent
question earlier in the
off the recording
changed their rnind
left
UV.O.\VV!,.H

The
resooitK!e~ms

2 instrument

test was """ ..'""-""'·"
from the 2011 test

consisted of

the 2012 test
at

have an impact on
a"''·"'"'uu1u n;:siJa•c'V~,.~T'C>r!

5

Person Asked

Person Cooperation Rate '""

' who were asked
Nuniher of persons
the consent question and agreed to
he recorded
the entire interuiew
persons
x
asked consent

mo

Full CAR! HH

Number of persons who agreed
to be recorded for the
entire interview
-N~l- f...
um .Jer o · comp l.etc and. proxy x 100
interviews
Partial CARI HH

Number of HH where all
respondents agreed to he recorded
for the entire interview
·····1riirnJ;e1:·of'"f:Yira:~Jiea·--

............ x

100

Rate=

Number of HH where some
respondents agreed to be
recorded for

consent question
Full CARI

lfff

HH Rate=

Nwnber

H fl respondents who
to be recorded
the
_ _..........,.entire interview
x 100
Numf}er of Hl:! respondents
asked consent

Number of HH where all
respondents did not agree to he
recorded
the entire interview
- ........,.......- .....................~.,........,_,,,....................c ..., ...,..,.....~-~.....,.........~··· X JOO

to
agreed or
not agree to
cooperation rates
(2008)

""'"If',..."'"''""''"

""""'·,,.·"·'"'""';,. ,, although some
rates between interviewers
be attributable to
the consent propensity modeling).

6

to
regions and
that are
likely to consent to being recorded
privacy concerns.
(2007) found
concerns Jed to
socioeconomic status, gender, health status, and
record linkage consent rates by
ethnicity. Persons with privacy concerns about record linkage may also have concerns
about their
being recorded
CARI, and thus these demographic factors will
be
for inclusion in the models.
household levels is used to explore
Logistic regression modeling at the person
whether person, household, and interviewer characteristics can explain variation in CARI
consent propensity. The models are not weighted. Sirkis (20 I tP0'11(1ents are
treatment group and asked
CARI consent
on response rates and quality measures.
of

Further research is needed to directly measure

VHL

Y. J. (2007). Record linkage
), 18.

17

and