Attachment 11 - Analysis Plan

Gemini Proof of Concept Test

Analysis Plan October 9, 2014

Methodological Issues
Concept	Analysis	Comparison Group	Evaluation Guidelines1
Research Objective 1. Feasibility of completing one wave of the proposed survey redesign
Feasibility of Design	Overall response rate: % HHs that complete recall, complete at least one diary and complete records interview	Production CEQ and CED rates; same counties as in the test	RR must be higher than 60%, or no lower than 10% of production
	Amount of data collected: total expenditures reported vs. production (diary + interview)	Production total expenditures, CEQ and CED; same counties as in the test	Total expn. must be no lower than 10% of production
	Quality of data collected: Percent interview and diary data that had to be allocated or imputed	Percent production interview and diary data that had to be allocated or imputed	Editing rates must be no higher than 10% of production
	Subgroup comparisons: Compare overall response rates, quantity and quality of data across demographic subgroups (gender, age, race, household income, household size)	Production CEQ and CED rates; same counties as in the test	No decision point
Research Objective 2. Effectiveness and costs of incentives
Overall effectiveness	% of HHs that reported receiving advance letter (with $2)	None	More than 50% of HHs say yes
	% of debit cards that were cashed	None	No decision point
Gaining cooperation	Overall response rate: % HHs that complete recall, complete at least one diary and complete records interview	Production CEQ and CED rates Individual Diary Feasibility Test (IDFT) response rates	RR must be higher than 60%, or no lower than 10% of production
Reducing contact attempts	Number of contact attempts prior to Visit 1, as indicated by time stamp of collection of HH Roster	Production number of contact attempts prior to Wave 1 Production average number of contact attempts prior to CED	Number of contacts attempts should be equal or less than to the average number of contact attempts required for wave 1 and diary placement
	Average total number of contact attempts	Production average total number of contact attempts for the first and second wave cases + diary cases	Number of contacts attempts should be equal or less than to the average number of contacts in production
Encouraging engagement	% of HHs that received recall incentive, diary incentive, full/partial records incentive	None	More than 60% of HHs were eligible for all three incentives
Research Objective 3. Respondent willingness to complete all components of the integrated survey
Overall Response Rates	Overall response rate: % HHs that complete recall, complete at least one diary and complete records interview	Production CEQ and CED rates (duplicate item)	RR must be higher than 60%, or no lower than 10% of production
	% of HHs that refuse one or more survey elements	Production CEQ and CED refusal rates	Refusal rate must be no lower than 10% of production
Recall Interview	% of HHs that refuse the recall interview	Production CEQ refusal rates for Wave 1	Refusal rate must be no lower than 10% of production
Diary	% of eligible HH members who complete diary	IDFT	At least half of eligible HH members complete (1+ entries) a diary; or were eligible blanks
Records Interview	% of HHs that refuse the records interview	Production CEQ refusal rates for Wave 2	Refusal rate must be no lower than 10% of production
Records	Number and type of records provided by each HH	None	At least 75% of participating HHs provide one or more records (as indicated by FR)
Operational Issues
Research Objective 4. Number/Effectiveness of contact attempts necessary to obtain a completed interview
Contact Attempts	Number of contact attempts prior to Visit 1	Number of contact attempts prior to first wave in production, and Number of contact attempts prior to production CED	Number of contacts attempts should be equal or less than to the average number of contact attempts required for wave 1 and diary placement
	Number of mid-week Contacts	IDFT contact rates	None, used to understand FR effort & R response across all parts of interview
	Effectiveness of mid-week Contacts in obtaining additional entries	Comparison of number of entries in individual diaries prior to mid-week contact to post mid-week contact; compare individual diaries with no mid-week contact to diaries with mid-week contact	None, used to understand the effectiveness of contacting the respondent during diary week.
	Number of contact attempts prior to Visit 2 as indicated by the time stamp for the first records CAPI section	None	None, used to understand FR effort & R response across all parts of interview
	Total number of contact attempts	Production total number of contact attempts for the first and second wave cases + diary cases	Total number of contacts attempts should be equal or less than to the average number of contact attempts required for wave 1 and diary placement
Research Objective 5. Length of the Interview Elements
Timing	Average time of recall interview	Average time of “recall sections” in production CEQ	Average recall interview is less than 60 minutes
	Average of each Recall Section	Average of “recall sections” in production CEQ	No decision point
	Average Visit 1 time		Average visit 1 is less than 60 minutes
	Average Visit 2 time, excluding debriefing		Average visit 1 is less than 60 minutes
	Average time spent at diary placement screen to determine time it takes to review instructions / length of diary placement		No decision point
	Length of Diary Recall Process		No decision point
	Total time of records interview	Total time of “records sections” in production CEQ	Average records interview is 60 minutes
	Time spent at record placement screen to determine time it takes to review instructions / records placement		No decision point
	Time of each Records section	Time of “record sections” in production CEQ; Time of “record sections” in production CEQ for Rs who used records “almost always”	No decision point
Research Objective 6. Technical issues respondents have with completing the online diaries
Reported Issues	Log of calls (and emails) to help desk and reasons for the contact
	Respondent comments during debriefing		No decision point
Observed Issues	Number of times Rs accessed the change password page but was not able to change their password		No decision point
	Total time R spent on change password page, broken down by successful password change and non successful password change
	Number of unsuccessful log-in attempts		No decision point
	Number of blank diaries, with and without successful log-ins		At least half of HH who were given a web diary members completed it (1+ entries); or were eligible blanks
	Item missingness (e.g. amount, characteristic)*		No decision point
	Number of items reported per diary and per HH, by category and overall*	Number of items reported by CU, by category and overall in production CEQ and CED	No decision point
	Amount of expenditures reported per diary and per HH, by category and overall*	Amount of expenditure reported by CU, by category and overall in production CEQ and CED	No decision point
	Number of receipts provided by HH with more than 5 items* (from FR debriefing)		No decision point
	Number of paper diaries completed by Rs given electronic diaries	Compare mode placed variable with the mode of most expenditure entries	No decision point
Non-Reported Issues	R issues identified by FRs through case notes and/or debriefing		No decision point
Research Objective 6a. Individual Diary Placement and Pickup Process
Diary Placement Process	Number of HH members present during placement (FR debriefing)		No decision point
	Number of DK responses for proxy placement questions		No decision point
	R debriefing questions about diary placement		No decision point
	FR debriefing questions about diary placement		No decision point
	Whether a login attempt was made during placement (FR debriefing)
Diary Pickup Process	Number of HH members present during pickup		No decision point
	Number of DK responses for proxy pickup questions		No decision point
	Percent of individual diary keepers who received an incentive		No decision point
	FR debriefing questions about diary pickup		No decision point
Research Objective 7. Per completed interview cost of administering the survey
	Total number of contacts per case		Total number of contacts attempts should be equal or less than to the average number of contact attempts required for wave 1 and diary placement
	Total interviewing time per case		Total interviewing time (Visit 1 + Visit 2) should be less than 180 minutes
	Total amount of incentives given per case		No decision point
	Total amount of incentives cashed per case		No decision point
	Other costs (travel, preparation) as available from Census		No decision point
	Total cost (interviewing time + incentives + other costs)		Estimate of total survey costs, excluding incentives, is within 10% of current production budget
Experience
Research Objective 8. Respondent Experience
Respondent Debriefing	Analysis of R debriefing questions		No decision point
Research Objective 9. FR Experience
FR Debriefing	Analysis of FR debriefing questions		No decision point

* Not definitive measure of technical difficulty

1 The proof of concept test goal is to measure the feasibility of implementation of the redesign plan as well as the improvement of measurement. The guidelines are a starting point for evaluating these goals. Looking at all of the guidelines as a whole will help determine the success of the design.

5