Supplemental Power Point

Influencing Mode Choice in a
Multi-Mode Survey

May 2012

AAPOR Conference Presentation

Geraldine Mooney  Cheryl De Saw   Xiaojing Lin

Andrew Hurwitz  Flora Lan

<number>

<number>

 

• • Potential Advantages 

• • • • –Lower data collection costs 

  • Potential Disadvantages 

    • • – Pushing sample members in a direction they don’t want to go, might negatively impact the response rate 

The Research Issue

<number>

<number>

<number>

 

• 2008 National Survey of Recent College Graduates (NSRCG) 

• • – Sponsored by the National Science Foundation 

  • –Conducted every two to three years since 1974 

  • –Sample:18,000 recent bachelor’s and master’s degree graduates  in the sciences, health and engineering   

• Increasing Data Collection Costs 

  • –Locating challenge 

  • –Difficult to motivate 

• 2008 NSRCG Mode and Incentive Experiment 

  • –Looked at the extent to which incentives and mode can be   used to increase web completes and/or response rates 

Why are We Interested?

<number>

<number>

<number>

 

Web Outcomes When Web and Paper Offered Simultaneously:

• Quigley et al (2000): In a sample of  active military personal, military wives, civilians and reservists, 23% responded by web  

• • –73% when web and paper were offered sequentially, although the response rate dropped slightly 

• Schonlau, Asch, and Can (2003): In a sample of high school graduates going off to college, about 1/3 responded by web  

• Millar and Dillman (2011): In a sample of college students, 53% responded by web when both email and postal reminders were sent  

  • –43% if only postal reminders were sent 

Web Completes in Multiple Mode Surveys

<number>

<number>

<number>

 

     Multi-Mode Approach : A Viable Means for Combating Declining

                                             Response Rates But Results Not Uniform

• • Groves & Kahn (1979), Tarnai & Paxton (2004):  Respondents have mode preferences; multi-mode surveys  

  • Dillman (2009), Mooney et. al. (2007):  Rather than increase response rate, additional modes migrate completes from one mode to another 

  • Millar & Dillman (2011):  “Modest” support for choice lowering response; when offered sequentially, web and paper are as effective as paper only 

  • Grigorian (2008): 2006 Survey of Doctorate Recipients offered sample members their preferred mode when possible, did not improve response  

  • Olson, Smyth, Wood (2010): In a mode preference study, regardless of mode preference, when offered first, respondents responded by paper 

Response Rates in Multi-Mode Surveys

<number>

<number>

<number>

 

Randomly assigned sample (17,851) into 8 groups defined by:

1. • Initial Response Mode: Two Alternatives 

1. • • •Web Only:  Paper questionnaire not sent until the 2nd survey mailing 

     • •Web/Paper:  Paper questionnaire sent in both the initial and 2nd survey mailings 

   • Incentive Amount (postpaid): None,  $20,  $20-$30 differential 

   • Timing of the Incentive: 1st mailing or 2nd mailing 

Compared web completes/response rates at 3 time points

1. • T1: Immediately Prior to Second Mailing (12/8/08) 

   • T2: At Start of CATI Follow-Up (12/17/08) 

   • T3: Six Weeks After Start of CATI Follow-Up (1/28/09) 

2008 NSRCG Incentive and Mode Choice Experiment

<number>

<number>

<number>

 

Groups	First  Mailing	Second Mailing	Sample Size
Web First
Group 1	No incentive	No incentive	3,569
Group 2	$20	$20	1,785
Group 3	No incentive	$20/$30 web	1,786
Mail/Web
Group 4	No incentive	No incentive	3,571
Group 5	$20	$20	1,784
Group 6	No incentive	$20/$30 web	1,786
Group 7	$20/$30 web	$20/$30 web	1,785
Group 8	No incentive	$20	1,785
Total			17,851

2008 NSRCG Treatment Groups

<number>

<number>

<number>

 

• Will limiting the initial response mode to web only: 

• • –Increase web completes over simultaneously offering both paper and web?  

• • • •Our expectation: Yes 

• • –Negatively impact the response rate in a multi-mode survey?  

    • •Our expectation: No 

Limiting the Initial Mode Research Questions

<number>

<number>

<number>

 

• Will Offering a Differential Incentive that Favors Web Completes: 

• • –Increase web completes more than no incentive or an incentive that rewards completes in any mode equally? 

• • • •Our expectation: Yes 

• • –Increase the overall response rate more than an incentive that rewards all completes equally? 

• • • •Our expectation: Yes 

The Differential Incentive Research Questions

<number>

<number>

<number>

 

Findings

<number>

<number>

<number>

 

Limiting Initial Response Mode to Web Only: Impact on Web Completes

<number>

Groups	N	Prior to Second Mailing	Start of CATI Follow-Up	After Six Weeks of CATI
Web First Total (1-3)	7,140	97.0%	95.3%	75.5%
Web/paper Total (4-6)	7,141	58.9%	59.4%	57.9%

<number>

<number>

 

Limiting Initial Response Mode to Web Only: Impact on Response Rates

<number>

Groups	N	Prior to Second Mailing	Start of CATI Follow-Up	After Six Weeks of CATI
Web First Total (1-3)	7,140	18.4%	22.0%	44.4%
Web/Paper Total (4-6)	7,141	19.7%	22.8%	44.3%
No significant difference between mode comparison groups

<number>

<number>

 

Web First Groups	Incentive	Mailing	N	Prior to Second Mailing	Start of CATI Follow-Up	After Six Weeks of CATI
3	$20-$30	2nd Mailing	1,786	98.0%	95.6%	79.9%
2	$20	Both Mailings	1,785	96.2%	94.8%	78.7%
1	None		3,569	96.8%	95.3%	71.6%

Differential Incentive: Web Completes in Web First Groups

<number>

<number>

<number>

 

Web/Paper Groups	Incentive	Mailing	N	Prior to Second Mailing	Start of CATI Follow-Up	After Six Weeks of CATI
7	$20-$30	Both Mailings	1,785	91.6%	91.1%	81.7%
6	$20-$30	2nd Mailing	1,786	56.8%	62.1%	64.7%
5	$20	Both Mailings	1,784	64.7%	64.7%	59.8%
8	$20	2nd Mailing	1,785	54.5%	56.3%	54.3%
4	None		3,571	57.0%	58.1%	53.6%

Differential Incentive: Web Completes in Web/Paper Groups

<number>

<number>

<number>

 

Web/Paper Groups	Incentive	Mailing	N	Prior to Second Mailing	Start of CATI Follow-Up	After Six Weeks of CATI
7	$20-$30	Both Mailings	1,785	91.6%	91.1%	81.7%
6	$20-$30	2nd Mailing	1,786	56.8%	62.1%	64.7%
5	$20	Both Mailings	1,784	64.7%	64.7%	59.8%
8	$20	2nd Mailing	1,785	54.5%	56.3%	54.3%
4	None		3,571	57.0%	58.1%	53.6%

Differential Incentive: Web Completes in Web/Paper Groups

<number>

<number>

<number>

 

Web/Paper Groups	Incentive	Mailing	N	Prior to Second Mailing	Start of CATI Follow-Up	After Six Weeks of CATI
7	$20-$30	Both Mailings	1,785	91.6%	91.1%	81.7%
6	$20-$30	2nd Mailing	1,786	56.8%	62.1%	64.7%
5	$20	Both Mailings	1,784	64.7%	64.7%	59.8%
8	$20	2nd Mailing	1,785	54.5%	56.3%	54.3%
4	None		3,571	57.0%	58.1%	53.6%

Differential Incentive: Web Completes in Web/Paper Groups

<number>

<number>

<number>

 

Table 5    Response Rates by Initial Response Mode and Type of Incentive
Group	Incentive	Prior to Second Mailing	Start of CATI Follow-Up	After Six Weeks of CATI
Web First
1	None	16.0%	17.9%	40.0%
2	$20/$20 both mailings	24.0%	29.1%	48.2%
3	None/$20/30	16.6%	23.1%	49.5%
Paper/Web
4	None	16.7%	18.8%	39.8%
5	$20/$20 both mailings	26.3%***	29.7%***	49.9%*
6	None/$20/30	19.3%	24.0%	47.7%
7	$20/$30 both mailings	28.6%	32.7%**	52.8%*
8	None/$20	17.9	22.6%	46.4%
Group 7  vs. Group 5  p< .05 at T2** and p <.1 at T3* Group 3  vs. Group 5  p< 0001 at T1,T2*** and p <.1 at T3*

Differential Incentive: Response Rate Comparisons

<number>

T

<number>

<number>

 

Table 5    Response Rates by Initial Response Mode and Type of Incentive
Group	Incentive	Prior to Second Mailing	Start of CATI Follow-Up	After Six Weeks of CATI
Web First
1	None	16.0%	17.9%	40.0%
2	$20/$20 both mailings	24.0%	29.1%	48.2%
3	None/$20/30	16.6%	23.1%	49.5%
Paper/Web
4	None	16.7%	18.8%	39.8%
5	$20/$20 both mailings	26.3%***	29.7%***	49.9%*
6	None/$20/30	19.3%	24.0%	47.7%
7	$20/$30 both mailings	28.6%	32.7%**	52.8%*
8	None/$20	17.9	22.6%	46.4%
Group 7  vs. Group 5  p< .05 at T2** and p <.1 at T3* Group 3  vs. Group 5  p< 0001 at T1,T2*** and p <.1 at T3*

Differential Incentive: Response Rate Comparisons

<number>

 

Table 5    Response Rates by Initial Response Mode and Type of Incentive
Group	Incentive	Prior to Second Mailing	Start of CATI Follow-Up	After Six Weeks of CATI
Web First
1	None	16.0%	17.9%	40.0%
2	$20/$20 both mailings	24.0%	29.1%	48.2%
3	None/$20/30	16.6%	23.1%	49.5%
Paper/Web
4	None	16.7%	18.8%	39.8%
5	$20/$20 both mailings	26.3%***	29.7%***	49.9%*
6	None/$20/30	19.3%	24.0%	47.7%
7	$20/$30 both mailings	28.6%	32.7%**	52.8%*
8	None/$20	17.9	22.6%	46.4%
Group 7  vs. Group 5  p< .05 at T2** and p <.1 at T3* Group 3  vs. Group 5  p< 0001 at T1,T2*** and p <.1 at T3*

Differential Incentive: Response Rate Comparisons

<number>

 

Table 5    Response Rates by Initial Response Mode and Type of Incentive
Group	Incentive	Prior to Second Mailing	Start of CATI Follow-Up	After Six Weeks of CATI
Web First
1	None	16.0%	17.9%	40.0%
2	$20/$20 both mailings	24.0%	29.1%	48.2%
3	None/$20/30	16.6%	23.1%	49.5%
Paper/Web
4	None	16.7%	18.8%	39.8%
5	$20/$20 both mailings	26.3%***	29.7%***	49.9%*
6	None/$20/30	19.3%	24.0%	47.7%
7	$20/$30 both mailings	28.6%	32.7%**	52.8%*
8	None/$20	17.9	22.6%	46.4%
Group 7  vs. Group 5  p< .05 at T2** and p <.1 at T3* Group 3  vs. Group 5  p< 0001 at T1,T2*** and p <.1 at T3*

Differential Incentive: Response Rate Comparisons

<number>

 

Conclusions

<number>

<number>

<number>

 

• Limiting the initial response mode to web only  

• • –Significantly increased web completes 

  • –No negative impact on response rate 

• The differential incentive  

  • –Significantly increased web completes among the Web/Paper groups 

  • –Modest impact on response rates  

• Of the 3 groups with the highest response rates, 2 offered differential incentives. Why might that be? 

  • –Using Barry Schwartz’s Paradox of Choice (2004) thesis, Millar and Dillman (2011) suggest choice increases cognitive burden, thus lowering response, especially if no compelling reason for mode choice is evident 

  • –A differential incentive, by rewarding one mode over another, provides a compelling reason, thus minimizing cognitive burden while rewarding respondents for choosing our preferred mode 

Our Main “Take Away” Points

<number>

<number>

<number>

 

• Use a sequential approach  

• • –Begin with web only  

  • –Introduce paper questionnaire in the 2nd mailing 

• Include a differential incentive in the 2nd mailing 

  • –Minimizes the cognitive burden associated with selecting a mode  

  • –Rewards respondent for using our preferred mode 

Best Practices for Influencing Web Completes

<number>

<number>

<number>

 

• Please contact: 

• • –Author 1 

• • • •gmooney@mathematica-mpr.com 

  • –Author 2 

    • •cdeSaw@mathematica-mpr.com 

  • –Author 3 

    • •ahurwitz@mathematica-mpr.com 

  • –Author 4 

    • •xlin@mathematica-mpr.com 

  • –Author 4 

    • •flan@nsf.gov 

For More Information

<number>

<number>

<number>

 

File Type	application/vnd.openxmlformats-officedocument.presentationml.presentation
File Modified	0000-00-00
File Created	0000-00-00

---

© 2024 OMB.report | Privacy Policy

---