2009 PFU Results 2

2

November 10, 2009

DSSD 2010 CENSUS COVERAGE MEASUREMENT MEMORANDUM SERIES #2008-D7-16

MEMORANDUM FOR: Magdalena Ramos

Co-chair, Census Coverage Measurement Operational

Integration Team

Patrick Cantwell

Co-chair, Census Coverage Measurement Operational

Integration Team

From: Gia Donnalley Signed November 10, 2009

Chief, Coverage Measurement Design for Data Collection Operations Branch

Decennial Statistical Studies Division

Prepared by: Travis Pape and Patricia Sanchez

Coverage Measurement Design for Data Collections Operations Branch

Decennial Statistical Studies Division

Subject: Lessons Learned from the August 2009 Census Coverage Measurement Person Followup Operational Test

This memorandum documents the lessons learned from the August 2009 Census Coverage Measurement (CCM) Person Followup (PFU) operational test based on trip report summaries and facilitated discussions with the participants in the test held on September 14 and 15 of 2009.

If you need further information, contact Patricia Sanchez on 301-763-9268 or Travis Pape on 301-763-5744.

Attachments

cc: DSSD CCM Contacts List

Lessons Learned from the Activities Related to the 2009 Census Coverage Measurement Person Followup Operational Test 2.0

1. Background

1.1 General

Due to budget shortfalls, the CCM PFU operation is one of the CCM activities that was dropped from the 2008 Census Dress Rehearsal. Although the PFU form and operation were tested in the 2006 Census Test, findings revealed that changes were needed to the PFU instrument and training prior to 2010. In order to assess these changes and determine if any further changes are required for the 2010 Census PFU questionnaire, the Decennial Statistical Studies Division (DSSD) conducted a reduced-scope field test in March 2009. Based on the results of the March test, new changes were made to the PFU questionnaire and an additional test was conducted in August 2009.

The PFU interviewing phase of the CCM program involves the follow-up of persons to resolve inconsistent information between data from the CCM Person Interview (PI) and from the Census. The CCM PI is designed to collect information about the members of each household selected for interviewing and about other places where each member lived or stayed around Census Day and around CCM PI Interview Day. After the CCM PI operation is completed, person matching is conducted between data from the CCM PI and data from the census. This involves a computer match, followed by a clerical match. Cases are targeted for a PFU interview when information is insufficient to determine residence, match, or duplication status. The completed PFU forms are reviewed and used to code a person’s final status.

The cases sent to PFU include people whose residence status is unresolved, CCM people who are possible matches to census people, possible duplicates in either the CCM or census, and CCM and census nonmatches. The interviewer must get as much of the address and dates of stay information as possible for each follow-up person in order to resolve each person’s status. Since 2010 PFU will be conducted about ten months after Census Day, some of the follow-up people may have moved, which would make it difficult to find knowledgeable respondents. A person is considered a knowledgeable respondent if they know the follow-up person well enough to answer questions about where the follow-up person in question was living during the year. If a person can’t be found who can answer these questions, three people who would have known the follow-up person if the person had lived in the area are needed before the follow-up person can be considered “fictitious” (that is, to classify the person as “did not exist in the area”).

1.2 Operational Test Design and Implementation

The main goal of this operational test was to test changes to Section A – Introduction, which includes questions to help the interviewer identify either a knowledgeable respondent who knows the follow-up person or three knowledgeable respondents who can verify that the follow-up person did not exist. In addition, we tested minor layout changes to the Cover Page, Person Questions in Section C, and the Record of Visits.

To conduct this test, we originally sampled 424 households from the recently expired sample of the Current Population Survey (CPS). We targeted people who had recently moved into or out of the CPS sample households. Due to insufficient name data, only 391 households remained. Of these, 286 cases were sent to the field (159 around Washington, D.C. in the suburbs of Virginia, and Maryland; 127 in Long Island, NY).

Interviewers for this test were eight Census Bureau Headquarters employees who were not familiar with the PFU operation. There were 11 observers who were also from Headquarters and the National Processing Center (NPC). Each interviewer was paired with an observer for each day of interviewing. The majority of the observers had participated in the development of the PFU form.

The PFU questionnaire was printed by Docuprint, which allows each questionnaire to be customized to each case, where different pages were included based on the type of case, and individual questions were customized.

1.3 Summary of Questionnaire Changes

Table A provides a summary of the different sections of the PFU forms from the first and second operational tests, and the changes.

Table A: Changes between the first and second operational tests

Section:	Change
Front cover: Contains the Census and PI roster	- Moved the introduction from Section A to the front cover
Section A: Introduction	- Major layout changes - Allowed interviewers to record names of all respondents instead of only those who were considered knowledgeable respondents - Removed Noninterview Assessment - Removed deceased response option to Question 1a - Removed detailed interviewer instructions (1c and 2b)
Section B: Possible Match	- No change
Section C: Person Questions	- Minor arrow changes - Added faint MM / DD / 2008 to Questions 1g, 2g, 3g, F1, F2, and 9d to guide interviewers to record exact date information - Added “Do not include short hotel stays.” above Questions 4 through 8 on page 3 of Section C - Removed name field within Question 9 - Reordered Question 10 to first ask any possible Miscellaneous Questions that the case might include - Changed Final-Person Level Outcome code from Complete (Unknown to KR) to Complete (Unknown to respondents)
Section D: April 1 occupancy	- No changes
Section E: Geocoding	- No changes
Back page: Record of visits	- Removed the following two columns: FR code and R type

2. Summary of Observations About the Questionnaire

This section compiles the lessons learned and suggested improvements for the PFU form based on the reduced-scope PFU operational test.

A. Cover Page:

1. There was a suggestion to rename Possible Reasons for Follow-up section to Why are we here? Interviewers were not thinking to use this section when respondents asked why they had to participate.

2. An observer suggested adding scripted reminders for the interviewers to ask for additional information regarding when household members would be home, when callbacks could be made, etc.

B. Section A:

1. There was a suggestion of adding a follow-up question to Question 1b asking respondents “who lived at the follow-up address on the reference date?”

2. In some instances, interviewers were confused about when to continue to Section C. An observer suggested adding, “You can ask Section C. Go to Next Page.”

3. Interviewers were not collecting respondents’ full names in Question 1c. A suggestion was made to include a faint print of First name and Last name within the response field of the question.

4. On one occasion, an interviewer interviewed about the wrong person. He/She realized this after checking the follow-up person’s age in Section A. We suggest changing the location of age in Section A and moving this information to the cover page.

C. Section C:

1. The faint print of MM/DD/2008 on all date questions (1g, 2g, 3g, F1, F2, 9d) was too dark and difficult for interviewers to write over to record information.

2. When there is not a miscellaneous question, the team decided the skip instruction should read, “Go to 10a” to avoid confusion regarding which question is omitted.

3. Interviewers requested adding check box to Question 10b to include the follow-up address. This was done in 2006 and mistakenly removed.

D. Record of Visits:

1. Interviewers felt the location of the Record of Visits section should be rethought. Having it at the end of the questionnaire caused some interviewers to forget about it and skip over it completely while interviewing. The team is adding this comment to the possible changes for the 2020 PFU Questionnaire.

3. Summary of Observation About Training

This section compiles the lessons learned and suggested improvements for the training for the 2010 PFU Operation.

A. Section A - Introduction:

• Training should spend additional time teaching interviewers proper techniques for probing.

• Training needs to include practice scenarios where interviewers need to use unscripted questions to find out additional information on the follow-up person(s).

• Interviewers requested practice scenarios with difficult respondents. This would allow trainees to practice dealing with reluctant respondents and open discussions for proper handling strategies.

• Training should encourage interviewers to inform respondents that the interview should only take a short amount of time – and starting interviewing immediately. Don’t offer the respondent an out by saying “Do you have time?” or “Can I ask you some questions?”

C. Section C:

• Training should have interviewers collect two addresses in Questions 4-7 and then remind them to make sure they do not skip Question 8 once they have completed Question I2.

• Interviewers felt training could provide them with additional verifying techniques to use in the field. Interviewers felt awkward having to ask questions which had already been answered previously.

D. Record of Visits:

• Training should spend more time on how to fill out the Record of Visits correctly, including how to record multiple proxy visits and no one home visits. Training should stress the importance of recording all attempts.

• Training should explain the proper use of Visit Outcome Codes, Final Outcome Codes, and Respondent Classification with various visit outcomes.

E. Other Training Comments:

• There was a suggestion to make the practice scenarios more realistic, such as making the interviewers stand and use the clipboard. It may be beneficial to have the interviewers pair up to practice scenarios instead of going through the practice cases as a class.

• One interviewer felt the flowchart in the training workbook could have been covered more during training. This interviewer felt that the flowchart would have helped her visually note differences in interview flow.

• Suggestions were made to increase the amount of trainee participation with questions asked by the trainer throughout the training in a Q&A format.

• An observer felt training should give instruction about how to clarify statements such as, “Not to my knowledge.”

• An interviewer requested that the training provide more information on the background of the PFU operation. This knowledge could help explain the reason for the visit to hesitant respondents.

4. Summary of Observations About Materials and Other

This section compiles the lessons learned and suggested improvements for the PFU procedures and concepts.

1. Materials – The majority of interviewers had positive comments about the training materials used.

2. Information Sheet – The group felt that the size of the GQ list and calendar could be increased. The writing on this information sheet was small, blurry, and difficult for some respondents to read due to the large margins that occurred because ACSD did not design the sheet.

3. Other - There were both positive and negative comments on using the clipboards with the PFU forms. Some interviewers felt a little overwhelmed with the amount of materials they had to deal with during the operation. Others felt appreciative to have the hard surface to write on while at the doorstep.

5. Recommendations

This section compiles the recommendations for changes to the PFU form and training.

A. Section A:

1. Switch the position of Print correct name if needed and Age to help interviewers locate the age of the follow-up person.

2. When the respondent answers that he/she knows who lived at the follow-up address on Census Day (1b), a new field should be added to collect the name of the person(s) who lived there.

3. Add faint First name and Last name to 1c to help interviewers remember to collect the respondents first and last name.

C. Section C:

1. The team has decided to change the MM/DD in Questions 1g, 2g, 3g, F1, F2, and 9d to the background color, similar to the dates recorded in the Record of Visits certification section.

2. Update the Docuprint message for Question 10 to read “Go to 10a.”

3. Add a checkbox to Question 10b to include the follow-up address.

D. Training:

1. The following concepts should be added to the practice scenarios:

   1. Movers/Proxies

   2. Fictitious person with three different respondents

   3. No one home

   4. Additional probing

   5. Multiple responses given in the same question

These should be added to existing scenarios where possible.

2. More instruction should be provided on proper verification techniques. Examples of good verification should be included in training.

3. Training should include examples of probes that can be used when talking to a respondent who is not knowledgeable to assist in finding a knowledgeable respondent.

4. Interviewer trainees should stand and use a clipboard during the practice scenarios to make the practice more realistic.