Attachment F - Updated NonResponse Bias Analysis

An Updated Nonresponse Bias Analysis of the Telephone Point of Purchase Survey:

Gabriela Arcos and Erik Bergmann

Based Upon Work by Patrick Falwell and Madeleine Saxton¹

July 22^th, 2013

1. Introduction

The Telephone Point of Purchase Survey (TPOPS) is a random digit dialed (RDD) computer-assisted-telephone-interview (CATI) survey that collects expenditure information from households, including the list of grocery stores, retailers, service providers, and shops where purchases are made. The survey is used exclusively by the U.S. Bureau of Labor Statistics (BLS) as the primary source of the retail establishment sampling frame for the Commodities and Services (C&S) Pricing Survey. A sample of the retail establishments (hereafter, outlets) reported in TPOPS is selected for participation in C&S, and prices of select consumer goods and services at these outlets are monitored for use in calculating the U.S. Consumer Price Index (CPI). The TPOPS is a rotating panel survey with each household participating over four consecutive quarters. The TPOPS is conducted by the Census Bureau for the Bureau of Labor Statistics (BLS).

This paper is an update to selected portions of the 2009 work, “Nonresponse Bias in the Telephone Point of Purchase Survey: A summary of three studies” by Patrick Falwell and Madeleine Saxton. It features amended tables and analysis, as well as preliminary results from a cellular phone frame, which began in the second quarter of 2012. Up until that point, respondents in the TPOPS were selected exclusively from a landline-based frame. Because of the declining number of landlines in the U.S, coupled with an increased prevalence of cellular-only households, a cell phone frame was added to the TPOPS. ² The addition of the cell phone frame was expected to mitigate possible bias in the survey resultsattributed to the underrepresentation of certain demographic groups within the landline-only frame.

TPOPS response rates are examined in Section II below. Section III compares demographics reported in TPOPS against the Census Bureau’s American Community Service (ACS). In Section IV, outlet frame characteristics are compared and contrasted for various cohorts of the total TPOPS sample.

2. Cellular and Landline Frame Results

Table 1 and Figure 1 show the response rate from the 1^st quarter in 1997 to the 1^st quarter in 2013. Response rates are calculated according to OMB’s response rate definition

Unweighted Response Rate³ = C a

C + R + NC + O + e(U)

Where,

C = number of completed interviews or sufficient partially completed interviews

R = number of refusals

NC = number of non-contacted sample units known to be eligible

O = number of eligible sample units not responding for reasons other than refusal

U = number of sample units of unknown eligibility, not completed

e = estimated proportion of sample units of unknown eligibility that are eligible,

(set to 0.27)

TPOPS response rates show a decreasing trend since BLS switched to an RDD/CATI survey in 1997. In the late 1990s, a response rate of close to 70% was achieved. By 2010, the response rate had fallen below 50%. From 2009 through 2011, response rates held somewhat steady, ending at 48.5% in the last quarter of 2011. With the introduction of the cell phone frame in the second quarter of 2012, overall response rates appear to have gone down, explained mainly by a lower response rate in the cell phone frame compared to the landline frame.

Table 1: Historic Response Rates by Quarter

Quarter	Frame	Interviews	Non Contacts	Refusals	Estimate of Unknown Population	Total	Response Rate
Q971	Landline	3,453	429	1,008	175	5,065	68.20%
Q972	Landline	6,330	940	1,736	357	9,363	67.60%
Q973	Landline	9,217	978	2,994	522	13,711	67.20%
Q974	Landline	12,086	1,266	3,800	741	17,893	67.60%
Q981	Landline	11,946	2,026	3,354	769	18,095	66.00%
Q982	Landline	12,000	2,419	3,363	748	18,530	64.80%
Q983	Landline	11,969	2,690	3,340	740	18,739	63.90%
Q984	Landline	11,996	2,312	3,601	911	18,820	63.70%
Q991	Landline	11,752	1,496	3,909	984	18,141	64.80%
Q992	Landline	11,514	1,284	4,034	793	17,625	65.30%
Q993	Landline	10,884	1,907	3,668	736	17,195	63.30%
Q994	Landline	11,502	1,667	4,019	738	17,926	64.20%
Q001	Landline	17,205	3,081	6,736	1,368	28,390	60.60%
Q002	Landline	16,666	4,004	6,455	1,303	28,428	58.60%
Q003	Landline	18,041	3,481	6,378	1,293	29,193	61.80%
Q004	Landline	17,904	2,320	6,094	1,300	27,618	64.80%
Q011	Landline	17,821	2,819	5,671	1,345	27,656	64.40%
Q012	Landline	13,680	2,357	4,731	963	21,731	63.00%
Q013	Landline	13,641	2,411	4,503	945	21,500	63.50%
Q014	Landline	13,824	2,290	4,424	936	21,474	64.40%
Q021	Landline	13,240	1,934	4,753	892	20,819	63.60%
Q022	Landline	13,166	2,559	5,144	997	21,866	60.20%
Q023	Landline	13,602	2,001	5,087	984	21,674	62.80%
Q024	Landline	13,684	2,178	5,136	955	21,953	62.30%
Q031	Landline	14,210	2,233	5,431	1,033	22,907	62.00%
Q032	Landline	14,084	2,920	5,019	1,016	23,039	61.10%
Q033	Landline	13,705	2,352	5,402	1,021	22,480	61.00%
Q034	Landline	13,759	2,132	5,618	1,076	22,585	60.90%
Q041	Landline	13,303	2,611	5,714	1,056	22,684	58.70%
Q042[1]	Landline	11,297	5,042	5,363	1,558	23,260	48.60%
Q043	Landline	12,643	4,364	4,664	1,535	23,206	54.50%
Q044	Landline	13,999	4,160	5,097	1,200	24,456	57.20%
Q051	Landline	14,245	4,562	5,223	1,271	25,301	56.30%
Q052	Landline	15,021	4,822	5,934	1,275	27,052	55.50%
Q053	Landline	14,961	4,796	5,808	1,226	26,791	55.80%
Q054	Landline	14,851	5,978	6,732	1,254	28,815	51.50%
Q061	Landline	15,009	6,983	7,825	1,338	31,155	48.20%
Q062	Landline	15,121	6,649	7,414	1,498	30,682	49.30%
Q063	Landline	14,939	7,532	5,857	1,489	29,817	50.10%
Q064	Landline	15,197	5,391	7,391	1,238	29,217	52.00%
Q071	Landline	13,276	7,130	6,553	1,365	28,324	46.90%
Q072	Landline	13,869	8,006	7,102	1,576	30,553	45.40%
Q073	Landline	13,538	7,332	7,032	1,386	29,288	46.20%
Q073	Landline	13,264	6,881	6,457	1,216	27,818	47.70%
Q081	Landline	13,207	6,506	5,898	1,101	26,712	49.40%
Q082	Landline	9,996	7,315	4,742	1,252	23,305	42.90%
Q083	Landline	11,507	5,992	5,458	1,125	24,082	47.80%
Q084	Landline	12,687	6,125	6,877	1,277	26,966	47.10%
Q091	Landline	13,083	7,487	6,193	1,370	43,144	46.50%
Q092	Landline	13,642	7,609	5,973	1,309	43,393	47.80%
Q093	Landline	14,107	6,842	6,293	1,329	43,988	49.40%
Q094	Landline	13,466	6,998	6,274	1,351	44,170	47.90%
Q101	Landline	13,761	6,586	6,139	1,400	43,996	49.40%
Q102	Landline	13,867	6,185	6,492	1,404	43,757	49.60%
Q103	Landline	13,705	6,024	6,773	1,317	43,526	49.30%
Q104	Landline	12,985	6,236	6,513	1,331	43,424	48.00%
Q111	Landline	13,434	5,860	6,066	1,324	42,940	50.40%
Q112	Landline	12,714	6,379	6,439	1,393	43,512	47.20%
Q113	Landline	13,057	6,461	6,209	1,199	43,984	48.50%
Q114	Landline	12,847	6,453	5,998	1,215	43,884	48.50%
Q121	Landline	12,765	6,308	5,970	1,249	43,303	48.60%
Q122	Combined	11,911	8,580	8,262	1,143	47,085	39.80%
Q122	Cell Phone	942	2,710	2,650	89	11,228	14.70%
Q122	Landline	10,969	5,870	5,612	1,054	35,857	46.70%
Q123	Combined	12,658	12,029	10,567	1,312	61,162	34.60%
Q123	Cell Phone	1,964	5,814	4,956	170	19,889	15.20%
Q123	Landline	10,694	6,215	5,611	1,142	41,273	45.20%
Q124	Combined	12,087	11,066	8,891	905	51,090	36.70%
Q124	Cell Phone	2,694	6,190	4,527	135	19,017	19.90%
Q124	Landline	8480	3881	4031	640.98	17032.98	49.8%
Q131	Combined	11748	10789	8942	812.43	32291.43	36.4%
Q131	Cell Phone	3,268	6,908	4,911	171.45	15,258	21.42%
Q131	Landline	9,393	4,876	4,364	769	32,073	48.40%

Figure 1

Table 2: Collection Results

Quarter	Frame	Interviews	Non Contacts	Refusals	Unknown
Q091	Landline	41.1%	23.5%	19.5%	15.9%
Q092	Landline	42.5%	23.7%	18.6%	15.1%
Q093	Landline	43.9%	21.3%	19.6%	15.3%
Q094	Landline	42.4%	22.0%	19.8%	15.8%
Q101	Landline	43.4%	20.8%	19.4%	16.4%
Q102	Landline	43.7%	19.5%	20.5%	16.4%
Q103	Landline	43.7%	19.2%	21.6%	15.5%
Q104	Landline	42.3%	20.3%	21.2%	16.1%
Q111	Landline	44.4%	19.4%	20.0%	16.2%
Q112	Landline	41.4%	20.8%	21.0%	16.8%
Q113	Landline	43.3%	21.4%	20.6%	14.7%
Q114	Landline	43.1%	21.7%	20.1%	15.1%
Q121	Landline	43.0%	21.3%	20.1%	15.6%
Q122	Combined	36.1%	26.0%	25.0%	12.8%
Q122	Cell Phone	14.2%	40.9%	40.0%	5.0%
Q122	Landline	41.6%	22.3%	21.3%	14.8%
Q123	Combined	31.6%	30.0%	26.3%	12.1%
Q123	Cell Phone	14.7%	43.5%	37.1%	4.7%
Q123	Landline	40.0%	23.2%	21.0%	15.8%
Q124	Combined	34.1%	31.3%	25.1%	9.5%
Q124	Cell Phone	19.4%	44.5%	32.5%	3.6%
Q124	Landline	43.7%	22.7%	20.3%	13.3%
Q124	Combined	34.1%	31.3%	25.1%	9.5%
Q124	Cell Phone	19.4%	44.5%	32.5%	3.6%
Q124	Landline	43.7%	22.7%	20.3%	13.3%
Q131	Combined	34.1%	31.3%	25.9%	8.7%
Q131	Cell Phone	20.8%	43.9%	31.2%	4.0%
Q131	Landline	45.2%	20.7%	21.5%	12.7%

• From 2009-2011, response rates appear to be stationary around the 50% mark. This trend continues in the landline frame from the second quarter in 2012 through the first quarter of 2013. Underlying this trend is a large decrease in non contacts which is slightly offset by a small increase in refusals, as shown in Table 2.

• At the time of this report, there are not enough data to analyze any long term trends in the cell phone frame, or the overall combined results. However, as evidenced in Table 2, the cell phone frame appears to have much higher refusal rates and non contact rates than the landline frame.

Table 3: Response Rate by Region 2005-2012

Area	Y2005	Y2006	Y2007	Y2008	Y2009	Y2010	Y2011	Y2012
East
Boston	52.7%	47.3%	45.3%	45.5%	46.5%	48.0%	49.5%	42.3%
Philadelphia	50.2%	44.0%	47.3%	48.5%	44.2%	48.3%	46.0%	42.0%
Pittsburgh	60.6%	56.8%	52.4%	52.5%	53.2%	59.4%	56.2%	45.8%
New York	39.3%	34.1%	29.5%	30.0%	33.2%	35.0%	33.9%	30.1%
NY-CT Suburbs	46.9%	44.1%	43.4%	42.8%	43.7%	47.4%	44.3%	40.0%
NJ Suburbs	49.9%	44.3%	42.0%	43.7%	42.6%	45.7%	45.7%	39.7%
X-Sized East	59.2%	44.9%	58.7%	55.8%	55.3%	56.6%	56.5%	47.7%
Average	51.3%	45.1%	45.5%	45.5%	45.5%	48.6%	47.4%	41.1%
Midwest
Chicago	51.5%	48.2%	45.4%	45.5%	44.8%	47.1%	45.2%	37.4%
Detroit	57.9%	55.3%	50.3%	50.4%	51.4%	51.4%	51.2%	40.6%
St. Louis	65.1%	56.5%	56.0%	56.2%	58.6%	54.8%	54.8%	38.0%
Cleveland	61.4%	56.7%	51.3%	54.3%	54.7%	52.6%	55.4%	45.3%
Minneapolis	61.7%	59.9%	58.1%	58.9%	58.6%	58.6%	60.1%	48.0%
Milwaukee	60.1%	59.0%	54.5%	58.0%	57.0%	60.4%	62.1%	51.7%
Cincinnati	60.8%	55.5%	58.1%	57.0%	54.9%	55.4%	55.6%	45.6%
Kansas City	61.6%	60.2%	58.2%	61.3%	58.2%	55.8%	61.8%	52.3%
X-Sized Midwest	63.8%	49.8%	58.5%	58.4%	58.9%	57.8%	59.3%	48.2%
Y-Sized Midwest	68.6%	61.7%	64.1%	61.7%	62.6%	61.3%	61.2%	45.8%
Average	61.3%	56.3%	55.5%	56.2%	56.0%	55.5%	56.7%	45.3%
South
Washington, DC	56.9%	51.3%	47.5%	49.0%	48.3%	49.3%	52.5%	44.7%
Baltimore	57.2%	55.3%	48.5%	48.1%	47.0%	47.9%	47.6%	39.2%
Dallas	55.3%	49.6%	44.1%	45.9%	44.7%	45.0%	46.0%	35.1%
Houston	50.2%	45.4%	42.9%	41.6%	38.8%	39.5%	37.9%	33.9%
Atlanta	44.7%	40.8%	40.2%	41.1%	41.4%	41.9%	43.7%	38.2%
Miami	37.7%	33.7%	29.9%	30.4%	33.4%	35.8%	32.7%	27.5%
Tampa	49.8%	45.9%	45.0%	46.7%	42.9%	46.4%	48.2%	41.8%
X-Sized South	55.5%	50.7%	47.5%	47.0%	47.6%	50.1%	49.8%	37.4%
Y-Sized South	59.2%	55.7%	50.2%	52.1%	50.5%	55.3%	51.6%	32.4%
Average	51.8%	47.6%	44.0%	44.7%	43.8%	45.7%	45.6%	36.7%
West
Los Angeles	44.6%	38.0%	35.8%	37.5%	38.4%	41.3%	36.8%	31.4%
LA Suburbs	53.3%	44.4%	40.0%	41.4%	43.7%	42.4%	45.1%	36.7%
San Francisco	50.6%	45.6%	42.9%	41.3%	42.2%	44.5%	42.7%	39.8%
Seattle	57.4%	53.2%	51.2%	50.3%	51.5%	53.5%	53.1%	44.9%
San Diego	52.8%	45.0%	41.7%	43.1%	43.2%	45.7%	45.2%	39.9%
Portland	62.6%	57.4%	56.1%	55.9%	57.4%	54.1%	54.4%	40.7%
Honolulu	46.3%	41.7%	41.7%	40.4%	41.4%	43.0%	43.9%	33.5%
Anchorage	61.4%	52.8%	51.4%	46.2%	46.5%	47.8%	45.7%	43.0%
Phoenix	55.1%	54.3%	48.4%	46.0%	47.2%	46.3%	45.6%	41.2%
Denver	59.0%	51.7%	52.2%	50.9%	51.5%	53.2%	53.2%	44.4%
X-Sized West	61.0%	48.7%	48.1%	51.4%	52.4%	51.6%	50.0%	39.4%
Y-Sized West	62.7%	62.2%	60.6%	58.2%	61.4%	61.6%	57.6%	41.2%
Average	55.6%	49.6%	47.5%	46.9%	48.1%	48.8%	47.8%	39.7%

• Historically, the response rate of the Midwest has been much higher than the response rates of the other three regions. In 2011, for example, Midwestern areas had an average response rate of 56.7%, while non Midwestern areas averaged 47.0%.

• The Midwest’s A-sized (self-representing) PSUs also had higher response rates than PSUs in other regions. In 2011, the Midwest’s A-sized PSUs had an average response rate of 55.8%, while the East, the South, and the West had average response rates of 45.9%, 44.1%, and 46.6%, respectively.

• Additionally, the average decline in response rate from 2005 to 2011 for the Midwest has been low, at 7.5%, compared to the West and South’s respective declines of 14.0% and 12.0%. Though at a lower absolute level than the Midwest, the East’s decline has also been slow, at 7.7%.

• New York City (A109) has traditionally had the lowest response rate, but was surpassed in recent years by Miami. The best performing PSU from 2009-2012 was Pullman, Washington.⁴

• The X-sized PSUs (metropolitan non-self-representing) and the Y-sized PSUs (nonmetropolitan non-self-representing) (53.9% and 56.8% respectively in 2011) had higher average response rates than A-sized PSUs.

• The introduction of the cell phone frame in 2012 has lowered response rates drastically on average.

Cell Phone Frame

Starting with the second quarter of 2012, the TPOPS dual frame design was introduced with the first panel of respondents. Over four quarters, additional panels were added to the cell phone frame until reaching its steady state sample size the first quarter of 2013.

Figure 2

Four key characteristics of the two frames (landline and cellular) and the respondents from each frame stand out:

• The cell phone frame contains better telephone numbers than the landline frame; the percent of eligible panel 1 cases is 57% for cell phone and 36% for landline.

• The cell phone frame includes a higher percentage of geographic screen outs, refusals and non interviews (grouped as eligible non interview in the graph).

• There are almost four times as many geographic screen outs in the cell phone frame than the landline frame.

• The sample inflation ratio (SIR) of cases that go out for interviews has held steady for the landline frame between 6 and 7 cases per interview. The cell phone frame fluctuated as the survey reached its steady state, and in the first quarter of 2013 the SIR was about 10.3 cases per interview.⁵

The cell phone frame is providing a quality frame of eligible respondents, but it is not as productive as the landline frame. It takes a larger cell phone sample size to reach the target number of interviews.

Summary

The TPOPS suffers from high nonresponse in both frames, however low response rates are not uncommon for RDD surveys. Furthermore, a survey with a low response rate but with respondents who are still representative of the target population suffers very little nonresponse bias. The low response rates in TPOPS, then, are only a problem if the nonrespondents differ from respondents. In the following section, we examine the demographics of TPOPS respondents as a judge for how closely they represent our target population.⁶

3. Demographic Comparisons to ACS

A comparison of TPOPS demographic data to the Census Bureau’s American Community Survey’s (ACS) demographic data is necessary to analyze potential non-response bias in TPOPS. The American Community Survey, an extension of the Decennial Census, is widely considered the most accurate estimate of the actual composition of the population, due to its breadth and extremely high response rate (97%).⁷

TPOPS collects limited demographic data from the respondent, allowing only some demographic comparisons to ACS estimates. The TPOPS demographic data are limited to the respondent, and not the other members of the household. The TPOPS respondents’ demographics are compared against the demographic data of ACS householders, from 3-year estimates limited to the same geographic frame as the TPOPS.⁸ Because TPOPS weights on non-demographic factors, the data below are weighted TPOPS percentages.⁹ The observed weighted number of households in each demographic category were then compared with the expected number of households from ACS percentages. Using a chi square goodness of fit test, the population estimates in TPOPS for all demographic groups studied were statistically significantly different than the ACS estimates, below an alpha of <.001. That is to say there is less than a 0.1% chance that the distributions are actually equal.

1. Race and Origin of Householder:

2008 2009

Table 4 Table 5

2008
Race of Householder	ACS Percentage	TPOPS Weighted Percentage
American Indian or Alaska Native	0.5%	0.5%
Asian	5.6%	2.9%
Black or African American	13.4%	8.0%
Native Hawaiian or Pacific Islander	0.1%	0.2%
Other	1.4%	4.2%
Two or More Races	5.5%	1.6%
White	73.5%	82.7%
Origin of Householder	ACS Percentage	TPOPS Weighted Percentage
Hispanic or Latino Origin	13.9%	8.8%
Not Hispanic or Latino Origin	86.1%	91.2%

2009
Race of Householder	ACS Percentage	TPOPS Weighted Percentage
American Indian or Alaska Native	0.4%	0.6%
Asian	5.6%	3.5%
Black or African American	13.5%	8.4%
Native Hawaiian or Pacific Islander	0.1%	0.3%
Other	1.5%	4.0%
Two or More Races	5.0%	1.7%
White	73.8%	81.5%
Origin of Householder	ACS Percentage	TPOPS Weighted Percentage
Hispanic or Latino Origin	13.9%	9.1%
Not Hispanic or Latino Origin	86.1%	90.9%

2010 2011

2010
Race of Householder	ACS Percentage	TPOPS Weighted Percentage
American Indian or Alaska Native	0.4%	0.5%
Asian	6.0%	4.4%
Black or African American	13.9%	9.5%
Native Hawaiian or Pacific Islander	0.1%	0.3%
Other	1.7%	4.2%
Two or More Races	4.6%	2.1%
White	73.2%	78.9%
Origin of Householder	ACS Percentage	TPOPS Weighted Percentage
Hispanic or Latino Origin	14.3%	10.1%
Not Hispanic or Latino Origin	85.7%	89.9%

Table 6 Table 7

2011
Race of Householder	ACS Percentage	TPOPS Weighted Percentage
American Indian or Alaska Native	0.4%	0.5%
Asian	6.1%	5.3%
Black or African American	13.9%	9.7%
Native Hawaiian or Pacific Islander	0.1%	0.4%
Other	1.8%	4.1%
Two or More Races	4.5%	1.8%
White	73.2%	78.2%
Origin of Householder	ACS Percentage	TPOPS Weighted Percentage
Hispanic or Latino Origin	14.5%	10.5%
Not Hispanic or Latino Origin	85.5%	89.5%

2012

Table 8

2012
Race of Householder	ACS Percentage (2011)	TPOPS Weighted Percentage
American Indian or Alaska Native	0.4%	0.5%
Asian	6.1%	4.5%
Black or African American	13.9%	9.5%
Native Hawaiian or Pacific Islander	0.1%	0.3%
Other	1.8%	4.6%
Two or More Races	4.5%	1.7%
White	73.2%	79.0%
Origin of Householder	ACS Percentage (2011)	TPOPS Weighted Percentage
Hispanic or Latino Origin	14.5%	9.6%
Not Hispanic or Latino Origin	85.5%	90.4%

Age Comparison

2008 2009

Table 9 Table 10

2008
Age of Householder	ACS Percentage	TPOPS Weighted Percentage
Under 35 years	20.2%	10.9%
35 to 44 years	21.5%	16.8%
45 to 54 years	22.3%	23.1%
55 to 64 years	16.8%	21.9%
65 to 74 years	9.9%	15.3%
75 to 84 years	6.9%	9.4%
85 years and over	2.5%	2.7%

2009
Age of Householder	ACS Percentage	TPOPS Weighted Percentage
Under 35 years	20.7%	8.9%
35 to 44 years	20.8%	16.4%
45 to 54 years	22.2%	23.4%
55 to 64 years	16.9%	22.2%
65 to 74 years	10.0%	15.8%
75 to 84 years	6.9%	10.1%
85 years and over	2.6%	3.2%

2010 2011

Table 11 Table 12

2010
Age of Householder	ACS Percentage	TPOPS Percentage
Under 35 years	20.4%	8.9%
35 to 44 years	20.1%	16.2%
45 to 54 years	22.1%	23.1%
55 to 64 years	17.5%	23.1%
65 to 74 years	10.4%	15.5%
75 to 84 years	6.8%	9.9%
85 years and over	3.5%	3.4%

2011
Age of Householder	ACS Percentage	TPOPS Percentage
Under 35 years	20.0%	7.7%
35 to 44 years	19.7%	14.9%
45 to 54 years	22.0%	22.5%
55 to 64 years	18.0%	24.8%
65 to 74 years	10.7%	16.4%
75 to 84 years	6.8%	10.3%
85 years and over	2.8%	3.4%

2012

Table 13

2012
Age of Householder	ACS Percentage (2011)	TPOPS Percentage
Under 35 years	20.0%	14.6%
35 to 44 years	19.7%	14.2%
45 to 54 years	22.0%	19.6%
55 to 64 years	18.0%	22.8%
65 to 74 years	10.7%	17.4%
75 to 84 years	6.8%	8.6%
85 years and over	2.8%	2.8%

Figure 3

Figure 4

Comparison of Frames

Table 14

	Landline	Cell	ACS (2011)	TPOPS Percentage
Under 35 years	6.1%	30.5%	20.0%	14.6%
35 to 44 years	12.5%	16.1%	19.7%	14.2%
45 to 54 years	20.5%	16.9%	22.0%	19.6%
55 to 64 years	24.7%	19.6%	18.0%	22.8%
65 to 74 years	21.3%	11.6%	10.7%	17.4%
75 to 84 years	11.5%	3.6%	6.8%	8.6%
85 years and over	3.4%	1.8%	2.8%	2.8%

Table 15

	Landline	Cell	ACS (2011)	TPOPS Weighted Percentage
American Indian or Alaska Native	0.5%	0.5%	0.4%	0.5%
Asian	3.8%	4.7%	6.1%	4.5%
Black or African American	8.6%	10.4%	13.9%	9.5%
Native Hawaiian or Pacific Islander	0.3%	0.2%	0.1%	0.3%
Other	3.1%	6.8%	1.8%	4.6%
Two or More Races	1.4%	2.1%	4.5%	1.7%
White	82.3%	75.4%	73.2%	79.0%
	Landline	Cell	ACS (2011)	TPOPS Weighted Percentage
Not Hispanic or Latino Origin	92.3%	88.4%	85.5%	90.4%
Hispanic or Latino Origin	7.7%	11.6%	14.5%	9.6%

Summary

As stated above, the differences between the TPOPS and ACS samples are statistically significant for race and age, for all years between 2008 and 2012, at an alpha <.001. The TPOPS race demographics are also significantly different from ACS, do not seem to be as systematically different as the age cohorts. The individual race cohorts appear to be systematically different than the ACS. The white, American Indian or Alaskan Native, and other cohorts were always overrepresented, while the black, Asian and multiracial cohorts were always underrepresented.¹⁰ The Hispanic cohort has also been traditionally underrepresented. With only 3 quarters of data, it remains to be seen what long-term effect the addition of the cell phone frame will have on these numbers. However, preliminary results show that it appears to mitigate these results.

The TPOPS has had a historic problem underrepresenting householders who are under 35 years old. Again, preliminary results from the cell phone frame show a marked increase in that percentage, as can be seen in Table 14. 2012 ACS data were unavailable at the time of publication, but when compared with the 2011 values, the cell phone frame appears to reach younger respondents, on average. Respondents 55 years of age and older are still overrepresented, but the effect seems to be mitigated. Since Falwell and Saxton also found the under 35 cohort to be the most systematic underrepresentation in TPOPS, close monitoring of this cohort is recommended as additional results from the cell phone frame and ACS become available.

As stated in the previous section, a survey can suffer from nonresponse bias if its respondents are not representative of the target population. The TPOPS sample is not perfectly representative of the target population. However, it should be noted that unless a demographic cohort differs markedly from others in its outlets and expenditures, under- or overrepresentation of a particular cohort in the TPOPS does not necessarily result in biased outlet sampling frames for the CPI. The next section examines these factors.

4. Outlet Frame Comparisons Between Demographic Groups

This section evaluates whether the coverage issues first identified in 2009 produce an outlet bias in the survey. We compare reported expenditures and outlet names from the dual frame by age groups, race, and frame, to analyze whether survey results could be biased due to an underrepresentation of the race and age cohorts discussed in section three.

• Hispanic Origin

• Race Black

• Age groups under 55 and over 55

The data analyzed include an average of 48,813 gross outlets reported per quarter. In both frames, each interview yields an average of 4 outlets.

Table 16

TPOPS Dual Frame Data by Quarter
	2012 Qtr 2	2012 Qtr 3	2012 Qtr 4	2013 Qtr 1
Respondents from landline	10,982	10,705	9,401	8,483
Respondents from cell phone	943	1,971	2,699	3,274
Gross outlet reported landline	43,817	42,774	37,636	34,775
Gross outlet reported cell phone	4,084	7,990	11,078	13,098
Landline outlet yield per interview	4.0	4.0	4.0	4.1
Cell phone outlet yield per interview	4.3	4.1	4.1	4.0

Data collected during these four quarters are used to perform this preliminary outlet bias analysis. In the cell phone frame, these four quarters of data represent only one fourth of a complete administration of the survey due to its split questionnaire design where subsets of questions are asked in each geographic area. Results for all categories in all areas will be available in 4 years.

Expenditure Levels

Six separate Wilcoxon rank sum tests were performed on reported Point of Purchase (POPS) category expenditures by frame and four different demographic characteristics¹¹. Figure 5 summarizes the number of categories within a major group found to have statistically different expenditures when tested by frame, age group, race, gender, or Hispanic origin ¹²

Figure 5

The high rates of statistically different expenditures were found in the major categories for Food and Beverages, Apparel, and Other Goods and Services where most of the data points in the shaded portion of the graph. More than half of the categories for Food and Beverages were found to be statistically different in five of the six groups. The race group for Black versus all other non White races is the only group where more Food POPs categories are similar than different.

The range of Food categories where expenditures were found to be statistically different included a variety of groups such as food away from home, spices, seafood, milk, and bread. The highest rates of statistically different food categories were between age groups under 55 and over 55, where 80% of categories were found to have differences in reported expenditures between groups. High rates of statistically different expenditures are also observed when evaluating groups by Hispanic origin, race, and frame. The high rate of statistically different expenditures for food categories may be driven by types of outlets that cater to specific demographic groups.

The next highest count of POPs categories having statistically different expenditures were categories for the major group Apparel. When testing expenditure differences between age groups under 55 and over 55 there are 48% of apparel categories found to have statistically different expenditures. When testing by Hispanic origin 52% of apparel categories had statistically different expenditures. This may be attributed to an income effect but this cannot be tested since TPOPS does not request respondents to provide information on household income.

Of the six tests run for age, race, gender, and Hispanic origin the tests of expenditures between age groups under 55 and over 55 had the most categories found to be statistically different. Not only were the majority of Apparel and Food categories found to be different but also Other Goods and Services, Housing, and Medical Care. Overall 54% of POPS categories were found to have expenditures statistically different between under 55 and over 55. This result is consistent with expenditure data collected from the Consumer Expenditure Survey which has found older cohorts have different spending patterns than younger cohorts. ¹³

Figure 6

Comparing the percent reporting by POPs Categories and demographic groups shows that Food categories are the most reported outlets across frames, age groups, and races. POPs Categories in the Medical care major group have the second highest percent reporting for cohorts that are over represented in the survey (older, white, and not Hispanic). Groups that are under-represented in TPOPS (younger, non-white, and Hispanic) on average have a higher rate of reporting for Apparel categories.

These differences in reporting likely contribute to the expenditure differences shown in Figure 5. Both results indicate there may be different shopping habits of the under represented groups.

1. Outlet Rankings by Demographic Groups

The previous section examined differences in outlet expenditures and percent reporting between demographic groupings, while this section examines the rankings of reported outlets. Before analyzing the lists of outlets reported, the data were cleaned to minimize variation in outlet spelling. ¹⁴ Using four quarters of data from the dual frame, the frequency of outlet names were ranked within each demographic cohort at the Major Group level¹⁵

• Top ranked outlets reported by Hispanic origin were compared to the list of all outlets reported by non Hispanic origin.

• Top ranked outlets reported by all non white race groups were compared to the list of all outlets reported by the race group for white.

• Top ranked outlets reported by the race group for black were compared to the list of all outlets reported by the race group for white.

• Top ranked outlets reported by the age group for under 35 were compared to the list of all outlets reported by the age group over 35.

• Top ranked outlets reported by in the cell phone frame were compared to the list of all outlets reported in the landline frame.

The results of the outlet overlap comparisons are summarized in Table 17 at the Major Group level for frame, race, Hispanic origin, and age group. In the list of top 10 through top 50 outlets for every group there is a high degree of overlap reporting across all major groups. Hispanic origin has the least amount of overlap in the larger list of top 100 and top 200 for most major groups. Similar results are observed when comparing the top 200 outlets reported by the race group Black to the race group White; six out of the eight major groups have less than 100% overlap.

Table 17

The final report on the TPOPS cell phone test included a similar analysis of outlet overlap on test data collected in the Consumer Expenditure Interview Survey (CE) in the second quarter of 2011¹⁶. The data from CE reported a high degree of overlap in the top 20 outlets between landline and cell phone households. These results were found to be inconclusive due to the small sample. The results from four quarters of TPOPS data also show a higher rate of overlap between cell phone and landline respondents when evaluated at the major group level across all PSUs. This result is also preliminary given only four quarters of data were available at the time of the study.

Do any of these differences impact outlet selection for the CPI? To answer this question, analysis was also conducted at the PSU/POPs category level which is consistent with the administrative design of the TPOPS survey. The dual frame is compared to results of the landline portion of the survey to measure the impact of adding the cell phone frame.

Collected outlet data from the two most recent quarters, 2012 quarter 4 and 2013 quarter 1, along with expenditure and weighting results from the dual frame design, are used to calculate outlet probabilities at the PSU/POPs category level for the dual frame and for the landline only portion of the frame. The outlets for the two groups are ranked using the weighted probabilities within a PSU/POPs category cell and truncated at the top 5 outlets. The two ranked lists are matched on outlet names to perform a comparison of top 5 to top 5 outlets. A ratio is calculated for the number of outlet name matches to the number of ranked outlets (5 in most cases) within each PSU/POPs category. The table below summarizes the average rate of matching outlets in the top 5 from the dual frame and the top 5 from landline frame. The average percent of matching outlets for Education categories in PSUs from the Northeast is 89%, the highest among the regions and major groups. The lowest percent of matching outlet is observed in the major group Medical Care in PSUs from the West, where on average there is 52% matching outlets in these cells. Overall the dual frame does provide a high degree of matching outlets with the landline frame at the PSU/POPs category level when evaluating the top 5 ranked outlets.

Table 18

Although similar outlets appear in the top 5, the position of those outlets by their rankings in the dual frame and the landline only frame are different. The next part of the analysis compars the outlet rankings across the two dual frame and landline frame at the PSU/POPs category level. Over all PSU/POPs category combinations (2065 cells), 669 cells have outlet rankings that are the same in only 10% of cases. In 262 cells (PSU/POPs categories) the top 5 ranked outlets are identical in the dual frame data compared to the landline frame data. The cells with the largest amount of differences in outlet rankings are Food categories, followed by Housing.

Table 19

The data in Table 18 reveal that across PSUs and POPs categories almost 70% of the same outlets appear in the dual frame data and the landline only data of weighted and ranked outlets. Table 19 shows that there are many differences in the actual outlet rankings of the top 5 outlets from the dual frame to the landline only frame.

5. Conclusion

The TPOPS still has trouble reaching respondents under 44 years of age, especially those under 35. However, it appears that in most cases, unique outlets and outlet expenditure do not appear to differ significantly between these and other age groups when comparing data across PSUs and at the major group level. The impact from the cell phone frame is only apparent when evaluating the ranking of outlets at the PSU/POPs category level. Furthermore, as outlined in Section II, the cell phone frame appears to somewhat mitigate the historic underrepresentation of younger respondents.

A potential concern is TPOPS’s underrepresentation of blacks and those of Hispanic origin. As shown in the previous sections, outlet reports and expenditures do differ in many cases between blacks and whites, and Hispanics and non-Hispanics. While no allowances were made for potential confounding variables (i.e. regional or income differences) in these analyses, the results still hint at a potential non-response bias in the TPOPS due to underrepresentation of these demographic cohorts. However, we cannot state conclusively that the underrepresentation translates to biased CPI outlet frames. Outlet sample sizes within each elementary item-area cell are relatively small, on average 2.3 outlets are selected per POPS Category-PSU cell. Even though the underrepresented groups do have different expenditure patterns, which result in different outlet rankings, it is not certain that their higher rates of nonresponse have an effect on realized sampling outcomes. This could be quantified in the future once more dual-frame data become available, via Monte Carlo simulations.

1 This paper is an updated version of “Nonresponse Bias in the Telephone Point of Purchase Survey:

A summary of three studies” August 4, 2009, by Patrick Falwell and Madeleine Saxton.

2 See, for example, NHIS estimates of cell-phone only households: http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless201212.pdf

3 Response rate is calculated according to OMB’s response rate definition.

4 For a complete list of response rates by PSUs, see the Appendix, Table A.1.

5 The sample inflation ratio calculated here reflects phone numbers which have gone through screening to remove business phone numbers and non working numbers as much as possible.

6 See AAPOR: http://aapor.org/AM/Template.cfm?Section=Cell_Phone_Task_Force_Report&Template=/CM/ContentDisplay.cfm&ContentID=3176

7 See http://www.census.gov/acs/www/Downloads/handbooks/ACSGeneralHandbook.pdf , for general information, and http://www.census.gov/acs/www/Downloads/handbooks/ACSFed.pdf for additional information.

8 ACS 3-year estimates were used due to the unavailability of more accurate 5-year estimates, and in favor of less accurate 1 year estimates. In the previous version of this paper, 1-year estimates were used, which were forgone this time in favor of accuracy. However, while the 3-year estimates are more accurate, they were unavailable in 2005-2007 (due to the ACS being a new survey), so analysis from that time period has been dropped from this paper.

9 TPOPS uses a variety of adjustment factors in weighting, including those accounting for probability of selection, household size, and non interview adjustment factors.

10 Falwell and Saxton found the black cohort to be overrepresented in 2006, however, this difference could be due to using the less accurate 1-year ACS estimates.

11 The Wilcoxon rank sum test is a non parametric alternative to the two sample t-test that does not make assumptions of normality and only requires a minimum of 5 observations in each sample.

12 The test results by non responses show a high degree of similarity in reported expenditures for all major groups, minimizing the concern that groups of non responders could be different than those who respond.

13 Cashell, Brian W. “A Seprate Consumer Price Index for the Elderly?” January 20, 2010. http://www.aging.senate.gov/crs/aging9.pdf.

14 The cleaning of outlet data was only performed on the names and not street address, so the analysis did not take into account attributes related to location.

15 The rankings are based off of the frequency of outlets reported, in the case where there’s a tie the same ranking is assigned. As result when listing the top 10 ranked outlets there might be more than 10 outlets if multiple outlets were reported at the same frequency.

16 Stockburger, Anya. Memo to OMB: Update on the Telephone Point of Purchase Survey cell phone frame planning. June 6, 2012.

40