MEMO-Lessons Learned from the Activities Related to the 2008 Census Coverage Measurement Initial Housing Unit Followup Mini-Operational Test

2

September 3, 2008

DSSD 2010 CENSUS COVERAGE MEASUREMENT MEMORANDUM SERIES #2008-D2-19

MEMORANDUM FOR: Magdalena Ramos

Co-chair, Census Coverage Measurement Operational

Integration Team

Donna L. Kostanich

Co-chair, Census Coverage Measurement Operational

Integration Team

From: Gia Donnalley (signed 9/3/2008)

Chief, Coverage Measurement Design for Data Collection Operations Branch

Decennial Statistical Studies Division

Prepared by: Lora Rosenberger and Graciela Contreras

Coverage Measurement Design for Data Collections Operations Branch

Decennial Statistical Studies Division

Subject: Lessons Learned from the Activities Related to the 2008 Census Coverage Measurement Initial Housing Unit Followup Mini-Operational Test

This memorandum documents the lessons learned from the 2008 Census Coverage Measurement (CCM) Initial Housing Unit Followup mini-operational test based on trip report summaries and a facilitated discussion with the participants in the test and CCM Housing Unit Field Data Collection Team members held on June 26, 2008.

If you need further information, contact Lora Rosenberger on 301-763-3447 or Graciela Contreras on 301-763-5284.

Attachment

cc: DSSD CCM Contacts List

Lessons Learned from the Activities Related to the 2008 Census Coverage Measurement Initial Housing Unit Followup Mini-Operational Test

1. Background

During Census Coverage Measurement (CCM) Initial Housing Unit Followup (IHUFU), interviewers collect additional information for addresses unresolved after the matching operations. The CCM IHUFU operation attempts to collect additional information that might allow a resolution of match codes for any differences between the independent listing results and the Census address list and also to resolve potential duplicates. The IHUFU operation also determines the housing unit/group quarters (GQ) status for living quarters flagged during the CCM Independent Listing (IL) operation. The questions included for each follow-up case vary depending upon the reason the address is being sent to follow-up. Interviewers contact a member of each housing unit (or proxy, as a last resort) to answer the questions identified for a given address. When applicable, they also update the location of an address on the CCM block cluster map created during the CCM Independent Listing operation.

Completed IHUFU Forms are subject to Quality Control (QC) wherein QC interviewers return to the field to check a sample of housing units in each block cluster to ensure the work performed is of acceptable quality. If the cluster fails the QC, then the QC interviewer reworks the entire block cluster.

Due to budget and cost constraints, the CCM IHUFU operation was dropped from the 2008 Census Dress Rehearsal (DR). Still, significant changes have been made from the 2000 to the current DR/2010 IHUFU questionnaire. Additional types of units are being sent to follow-up, including possible GQs and matches to surrounding blocks. (A copy of the 2008 questionnaire is provided in Attachment A.) Also, the QC is significantly different from 2000. The changes include a modification to the QC sampling so every cluster is QCed and the QC interviewer must make a determination on whether a cluster passes or fails QC. The table below maps the 2000 questions to the 2008 questions and describes the changes to the questions.

2000 Question	2008 Question	Question Change
A	B	Responses have been expanded from 2000.
B	A	Question has been modified to ask about the block instead of non-shaded area of the map.
C	C	Responses have been expanded from 2000.
D1	D	Question has been modified.
D2	F1	Question has been modified.
D3	F2	Question has been modified.
E1	E	Question has been modified.
E2	G1	Question has been modified.
E3	G2	Question has been modified.
	H	New for 2008 as it is the first time possible group quarters were listed.
	I	New for 2008 as it is the first time possible group quarters were listed.
	J	New for 2008 as it is the first time possible group quarters were listed.
	K	New for 2008 as it is the first time possible group quarters were listed.
F	L	The response categories have been modified.
	M	The analyst question was present in 2000 but not as an item number. Its position and presence on the IHUFU form is new.
G	N	The question wording has been modified.

It is important to note that the IHUFU operation had not been previously tested in the 2010 Census lifecycle because the CCM Program was excluded from the 2004 and 2005 Census Tests and only the Person Phase operations of the CCM (including Person Interview, Person Matching and Person Followup) was included in the 2006 Census Test due to budget constraints. Therefore, the first test for the CCM Housing Unit data collection and matching activities for the 2010 Census was to be conducted in the 2008 Census DR, but since this was also descoped from DR, DSSD sponsored instead a reduced-scope field test for IHUFU. This mini-IHUFU test was conducted in order to assess the appropriateness of the questionnaire changes from 2000 listed in the table above and to determine if any additional changes are required for the 2010 Census IHUFU questionnaire and for the 2010 IHUFU interviewer training and procedures. It should be noted also that had the IHUFU Operation been conducted as part of the DR, some of the lessons learned from usual training development classroom exercises would have been implemented before going to the field in DR. However, because of the required quick turnaround from approval to do the mini-IHUFU test and its implementation, it was not possible to incorporate mini-test training lessons learned before the mini test field operation. Although it appears as if numerous changes are being recommended, many of these are in fact enhancements particularly about training. Overall the basic data collection approach for the housing unit information is sound. This document incorporates lessons learned from both training and field activities tested.

The mini-IHUFU field test was conducted from June 9, 2008 through June 20, 2008 in the Dress Rehearsal sites, San Joaquin County, California, and South Central North Carolina. In North Carolina, eight interviewers completed 29 clusters and partially completed two clusters. In California, seven interviewers completed 29 clusters and partially completed two clusters. Due to time constraints only 15 of the 29 completed clusters in North Carolina and 19 of the 29 completed clusters in California were QCed. Due to time constraints, an additional change had to be made to the planned QC; clusters that failed QC were not reworked. The clusters selected for the mini-IHUFU operational test were clusters that would have gone for follow-up had the operation occurred. However, the clusters containing the following type of housing units were targeted for this test as they contained the most changes from the 2000 questionnaire:

• Possible Group Quarters

• Two duplicate addresses (CCM and/or Census)

• Possible matches

• Analyst Special Question

• Matches to a surrounding block

CCM Regional Managers traveled to Headquarters (HQ) and the National Processing Center (NPC) to train HQ and NPC staff to be interviewers for the mini-IHUFU and QC operations. Support staff or employees unfamiliar with the IHUFU operation were recruited to be interviewers. A qualitative interviewer, that is, someone familiar with the IHUFU questionnaire and the goals of the operation, including staff from Decennial Statistical Studies Division (DSSD), Decennial Management Division (DMD), and Field Division, accompanied each interviewer. The qualitative interviewers observed the operations, reported on what went well, and identified any areas of concern with the instrument, training, or materials. In addition, interviewers QCed other interviewer’s work to test changes that had been incorporated for the IHUFU QC since 2000. The changes include a modification to the QC sampling so every cluster is QCed, and the QC interviewer must make a determination on whether a cluster passes or fails QC.

Note that although the test followed essentially interviewer field operations as planned for the cancelled Dress Rehearsal (and 2010) IHUFU, we did not have the field data collection control systems in place. Therefore, no field test of these control systems were undertaken. Neither was the work of other field staff (crew leaders, assistants, etc.) tested.

Following the completion of the field test, the completed forms were keyed into a spreadsheet at the NPC for analysis. In addition, the forms will be used to conduct a mini-operational test of the Initial Housing Unit After Followup Clerical Matching Operation in July 2008. This operation may yield additional information on ways to improve IHUFU for 2010.

This document details the lessons learned from the field test. As it was a small-scale test, it should be noted that not all recommendations would be incorporated for 2010. Future discussion will be needed to determine how to proceed with some of the recommendations.

2. Training

This section compiles the lessons learned and suggested improvements from the training sessions held at headquarters and the NPC for the operational test.

1. Reference List training - More training is needed to orient the interviewer on the use of the Reference List, including an explanation that units needing followup are identified with a “Y” in the FU (Followup) column. Emphasis should be made on how the CCM Initial Housing Unit Reference List is a very useful tool, especially when unable to locate an address for follow-up as it provides map spot numbers (MSN) for both the CCM and Census maps. Aside from orienting the interviewer around the cluster, the Reference List should be used to search for matches and duplicates whenever additional information is found, either on the ground or from a respondent. Examples in the training had the interviewers only look at the Reference List answering "nope, not here" to having the address listed on the reference list.

2. Match Code Training - Need to explain match codes in training (refer trainees to Appendix B of the Interviewer Manual), to help with what information we are trying to find out, and what caused the case to be included in IHUFU with this match code. Appendix B of the Interviewer Manual should be revisited to see if it can be simplified (less text) to make it more user friendly.

3. Forms Training - More training is needed on each type of follow-up form and what information is being collected for that type of follow-up case. The use of the letters and numbers in the questionnaire should also be explained more. This would minimize the amount of time the interviewer needs to “study” the follow-up form to determine what information is needed.

4. Map training - More time should be spent on reading and reviewing the different types of CCM and Census maps, including the purpose of having surrounding block maps. The use of maps to help locate clusters or an address within the cluster should be explained better. More exercises should be provided on locating and matching MSNs between the CCM and Census maps. Need to explain what sketch maps are.

5. Map spotting training - More training is needed on how to add map spots when needed to the block cluster map, including adding more than one map spot for a multi-unit Basic Street Address (BSA) with more than one building. The training only briefly mentioned that the number of units should be added in parentheses beside the map spot. Need to explain to interviewers about the Within Map Spot Numbers (WMSNs) – where it comes from, why it is significant, how do they use it.

6. Surrounding Block matches - More training is needed on how to determine block boundaries. For surrounding block matches, the concept of in the block versus outside needs to be explained more.

7. Multiunits - Emphasize in training the procedure for contacting a manager at a multiunit, what unit designations are, and WMSNs. Need to explain that multiple building multiunits may have a separate map spot for each building.

8. Definitions - More emphasis on housing unit definition and what a GQ is.  More training on how to determine future construction versus vacant lot (i.e., stakes and/or electric hookups present on site). Page 2-18 in the Lister's Manual explains quite well the difference between future construction and the vacant trailer lot NOT in a park. [However, someone observed a vacant trailer lot NOT in a park that was listed in CCM, but should not have been; stress in training that only vacant trailer sites IN A PARK should be listed.

9. Quality Control

• The QC practice exercises and scenarios in the field training materials should use the same geography as in the IHUFU interviewer production exercises and scenarios.

• Should remove redundant training content from the QC (safety, wrap-up, e.g.). If needed, can remind trainees but don’t need the entire sections.

• QC may need additional practice exercises, especially on critical errors. Also, need to explain purpose of QC better.

10. General Training Comments

• Add indexes to manuals.

• Include all map materials inside the workbook, not just the ones that are required for the practice so the trainee becomes familiar with all maps. Can the workbook pages be perforated? Trainees could remove maps and reference lists – and for QC what’s appropriate – before beginning practice. This would eliminate page turning, losing your place, and simulate field work.

• Need more emphasis on the importance of including notes to explain unusual circumstances.

• Explain during training what is in the IHUFU packet.

• Need to provide more training on explaining to the respondents why they may have already been visited by Census. Can a handout be created? A handout could allow us to explain how the Census/CCM operations work and why they may receive multiple visits. If a handout were created, it would need OMB Clearance.

• More training is needed regarding probing. We can’t just tell the interviewers that they need to probe if they don’t get all the information they need. Initially they don’t even know what all the information they need is, let alone how to elicit it.

• For this operation, where we are asking interviewers to be more detectives than interviewers, it would be helpful to teach the interviewers more about the purpose of the job and the big concepts they need to know to do it right. Training should emphasize the need to look at the big picture (goal of follow-up situation) when there are several cases in an apartment complex, reviewing all materials including the form, maps, and reference list, and orienting themselves with all units needing follow-up. Expecting our interviewers to think is not a bad thing, but they need more guidance during training about how we want them to think or what the intent/basis/reasons are behind the thinking they have to do.

• Include information on why the case is in the IHUFU workload – something like Address Canvassing and Independent Listing happened, NPC tried to match addresses (using addresses and map spots), those that couldn’t be matched or matched with certainty are sent to the field in IHUFU to see what is out there – are units a match, did CCM and/or Census miss or double count units on the ground, etc.

• The training tells the interviewer that they can reword the questions if they think the respondent will understand it better in different words.  This makes it very easy for the entire questionnaire to become a conversational interview, which quite few did. This training statement may have confused some of the trainers as well as the interviewers because most training that they may have had earlier instructed them to read the questions as worded. It has been noted that many of the interviewers initially did read the questions as worded but after a couple of interviews, they reworded the questions. This may have helped in obtaining responses, but by instructing the interviewers to reword, the question’s intent may have been lost. The training should be revised to eliminate providing the option of rewording the questions.

• Explain the header address information on the top of the form more.

• Should mention not to touch mailboxes, and mail.

• A few times a “don’t know” or “does not exist” answer for verifying a housing unit address would have been more appropriate than the “cannot locate” answer. Since the answer options were purposefully eliminated on the form, training should show the interviewers how to proceed.

• There is a question that asks, "how many HUs are at this BSA,” and newly trained interviewers answer this question with a "1" when referring to an apartment in a complex rather than with the actual number of units within the whole apartment complex.  Need additional training/emphasis on what is a BSA versus multiunit addresses, versus the specific unit.

• There was a lot of skipping back and forth in the manual through the training, causing confusion and disorientation.

• The importance of filling out the IHUFU form when the interview is completed, and not at the end of the day at home should be stressed during training.

• Where to enter the answer for the 'Special Question' (a question usually sent to follow-up by the Clerical Matching Analysts) was hard to find in the manual. A suggestion is to place under the 'Special Question' subtopic, instead of the 'Notes' subtopic.

• It was difficult juggling all the papers (IHUFU packet, maps, reference sheet). The materials (maps, reference list, and IHUFU forms) were quite bulky when working with them in the car. This is the nature of the job. Perhaps a mention in training to prepare the interviewers would be appropriate.

11. Scenarios/Examples

• Consider a training scenario on each Match Code, or the most common ones (if known).

• Explain and do a scenario to really use the map and Reference List (placing emphasis on the importance of the Reference List) to determine the status of units and if a match, for what is on the ground. Try to incorporate some of the weirder situations interviewers will find, will need to discuss. Perhaps a scenario where two units appear to be a match, then go to the map to show the comparison of the CCM map spot versus the Census map spot, explain why these addresses weren’t matched during clerical matching.

• Emphasize that interviewers could have two forms for the same unit in the packet. They should be told to look for more than one follow-up form in the packet to avoid a return visit for the second form. Perhaps a scenario could be added or review of the Reference List, to see a Census non-match case at the end that may be a match to a CCM unit on the list.

• Give a better explanation of what a block consists of, more specific example, as well as a visual example of a map and determining the boundaries.

• An example of different housing units in different blocks having the same MSN should be an example in the training. Different housing units can have the same MSN when there is a match in a surrounding block.

12. Possible Deletions from training –The assumption is IHUFU interviewers did NOT attend lister training and must be taught how to use maps, map spotting, etc, so this stays in training. However, there was a lot of time spent on measuring a map and the use of the compass on a map that are general map skills, not CCM concepts, and should be shortened.

13. Training Material Comments:

DX 1374 Interviewer Manual:

• Page 4-3: "Possible Match" and "Under Construction” definitions are incorrect.

• Page 4-11: Third bullet under CCM Address WMSN: replacing the term "single-

housing unit" with "individual housing unit” would make the difference between

that definition and the subsequent "single-family residence" less confusing.

• Appendix Page A-4: On title for Question A, should delete "CCM Address " since this can also be for a Census Address.

• Appendix A-19: The explanation under question code J, "When and how to ask this question" should be “This question is asked when respondent's answer to Question I was L: None of the above ".

DX-1374.1A Interviewer Workbook

• Page 1: Box in center of page with Map Spots 24 & 25 covers address below.

• Packet Cover Page Introduction: It was suggested that the Introduction include

"from the U.S. Census Bureau" rather than "from the Census Bureau".

• Workbook Pages 12 - 13: This scenario should not be used because it is something that could not occur.

DX1374.1 Interviewer Training

• Page G-55: Final Paragraph should be "question 5" not "question 4".

• Page G-59 Q3: map spot should be 30 (5) with explanation about including

number of units. Q5: need location descriptions for apartments 3 & 4 since

respondent initially said they have no numbers and both are on second floor.

• Trainees are instructed to read in Handbook Topic 2: Safety on p. 5-3, actually

5-2 to 5-4.

• Trainees are instructed to read in Handbook what to do if you are involved in a

vehicular accident while working on p. 5-5. This subtopic is on pp. 5-5 – 5-6.

DX-1333.1 QC Checker Workbook

• Incorrect Maps and Map spots pages 24 & 49. Errors on Reference List page

25.Too much information pre-filled on all forms, especially a problem with page 45.

QC Training

• Delete all references to Relisting in verbatim training and manuals since the Relisting Operation is dropped from the 2010 CCM program.

• Practice Exercises – Should also illustrate the Census maps.

• Practice Exercise 1 – Sample Case 1 – Question 9's MSN should be 4A.

• Practice Exercise 2 –Sample Case 3 – A critical error is in Item A (Level 1), an

incorrect checkbox is marked. There is no need to check for critical errors in Item

A1 as mentioned in the verbatim training text.

3. Procedures and Concepts

This section compiles the lessons learned and suggested improvements for the IHUFU procedures and concepts.

1. The use of “ok” on the IHUFU form during/after Dependent Quality Control (DQC) needs to be explained better. Does it mean that the cluster’s been through DQC, or does it mean that it passed DQC?

2. Map spotting business addresses and GQs in IHUFU will not occur in 2010 as this caused problems when implementing the mini-test. During the mini-test training, the interviewers were instructed to map spot businesses and non-housing units, which is contradictory to what listers did during IL. It was decided to keep the questionnaire as is for the mini-test and see how it was implemented, then revisit the concept of listing housing units only after the test. In 2000, businesses and GQs were not mapspotted during IHUFU. Most mini-IHUFU interviewers questioned why they were being instructed to add non-housing units due to the instruction, “Add new map spot to the CCM Block Cluster Map, then enter the map spot number here,” in a particular question. After discussion with observers of the mini-IHUFU test it was decided not to map spot these in 2010. If businesses and GQs were added to IL maps, the maps would need to be updated by analysts at NPC, deleting businesses and GQs, causing additional unneeded shipping and updating of CCM maps. This procedure will be revised and corrected for 2010.

3. If a multiunit has a manager/landlord office off site, could the lister (for IL) and/or interviewer (for IHUFU) call or visit the offsite managing offices?

4. For huge clusters, would it be okay for the interviewer to add notes that refer to a note on another form?

5. The distinction between units demolished and unfit for habitation can be gray. More guidance should be given on deciphering the two.

6. In rural areas, it would be very beneficial to allow the managers to send two enumerators at once so they could finish interviewing efficiently and in less time. It would have taken far longer to finish the clusters if there were only one of us...over twice the time (we estimated about 3-4 days instead of one). That would mean additional travel costs and salary costs.

7. Determine the procedure for adding a MSN. Should interviewers always use an alpha attached to the map spot they are inserting the address after, or for MSNs outside the cluster should they use the next highest number MSN?

8. If a respondent refuses, do we try for a proxy in 2010?

9. Need to discuss the instructions for classifying a unit that is presently vacant, for sale, zoned for commercial use, but was last occupied as a HU. Even though it is a special situation, it is not currently addressed.

10. Need to discuss if an address does not exist, should the FLD delete these MSNs or somehow mark on the map? If the address does not exist the analyst needs to know why. Does the address actually exist outside the cluster or does the number range exist in the cluster and the house number is not a valid house number. This makes a difference between making a possible address correction or coding it a geocoding error.

11. Need to discuss possibility of showing block cluster locator maps to respondents to determine non-visible boundaries for clusters, as they do not contain Title 13 data.

12. Need more emphasis on mulitunits, and to go to the building manager for a map of the complex, then use the map of the complex to figure out the block boundaries.

13. This is a labor-intensive operation. Note that while the actual interviews may take one to two minutes, and the paperwork an additional few minutes, the legwork to actually determine the information needed (finding the unit, determining which housing unit, which block, etc.) may take much longer. To complete a follow-up case, the times vary depending on the situation anywhere from 1-30 minutes on average.

14. Post-it notes are very useful and it should be considered supplying them to the field staff.

15. Need to discuss if a procedure to collect an auxiliary map at apartment complexes should be included in IHUFU.

16. Need to discuss if there are duplicate MSNs on the CCM map, how do the addresses get deleted from the CCM column of the reference list?

17. In QC, most critical errors were due to the confusion over what was considered a housing unit versus a GQ or the confusion as to the number of housing units at a multiunit (unsure whether the question was asking about the BSA or the housing unit). In order to clear up this confusion, more emphasis should be placed on the use of and interpretation of BSA.

18. Should emphasize for QC if cannot locate a specific follow-up unit, should read what the interviewer had put in notes for that case, as well as other forms near that

follow-up case.

19. Need to add procedure of rectification and QC check for a multiblock cluster.

20. Further discussion is needed on when to add a map spot, (i.e., Business or GQ).

4. Forms

This section compiles the lessons learned and suggested improvements for the IHUFU Form.

A. Initial Housing Unit Followup Cover Page:

1. Put the place name, zip code, and/or city on the cover page.

2. Need to discuss where the introduction is located. Having to flip to the cover page on a clipboard is difficult. Could be printed on the beginning of each follow-up form or on an extra sheet rather than the bottom of the cover page.

3. The locating instructions on the cover page were not used during most observations. Need to discuss if needed for 2010.

4. The IHUFU introduction needs some editing. It says, "Here is some information about what we are doing." Then the interviewer hands the respondent a Privacy Act Notice. The Privacy Act Notice states that the respondent's answers are confidential, it does NOT give information about IHUFU.

B. Housing Unit Reference List:

1. Place Census only cases in the approximate address order. Need to discuss how, but may be able to use Census ID order (when available), and/or insert Census only cases at the end of the listing for a specific street name when the street name is present. If the Census only case is lacking a MSN or street address, it needs a special code in BFU to keep it from being sent to the field.

2. Need to discuss having an indicator (asterisk, “Y”, or highlighting) for cases needing follow-up.

3. The reference list did not contain the number of units under the WMSN for a multiunit.

C. Initial Housing Unit Followup Forms:

1. The skip patterns need to be corrected for 2010. Skips should be implemented so things like businesses and GQs are not added to the maps to be later deleted. Non-housing units should be directed to another question that does not contain the map spotting instruction (as mentioned in the second bullet in Section III). Also, the skip pattern instructions should stand out better. Some suggestions are to place a box around the skip pattern instructions, or italicize the instructions in larger font. On the NE/DE form (for nonmatched or possibly duplicated E sample cases) why skip to question 6 if they are the same unit to ask the number of units question twice? Also, it is odd to ask the questions about the primary address when the duplicate has the correct address.

2. Since the forms are stapled together in clusters, it was cumbersome to figure out when a case started and stopped. There is a (continued with an arrow), but nothing signifying the case is on the last page. Would it be possible to place (stop, go to next) on the last page?

3. Need to discuss where the analyst’s special question should be on the form. It did not work well. In 2000, the question was not numbered, but was at the top of the form.

4. There were some possible matches where neither the CCM nor Census address was correct. It should not happen very often, but we need a new category for “Yes, but both the CCM and Census address is incorrect. Write correct address here:”

5. There were a few cases where the status of the housing unit fit multiple categories. Need to discuss marking all categories that apply and explain in notes.

6. The “no housing units at this BSA” response category was missing from the L question on Forms (NE) and (NI).

7. Need to discuss the M* cases (matches needing more information) being split into two forms, one for geocoding and one for unit status verification. Geocoding cases do not really require a respondent.

8. From observation, Question A answer, “Yes but address needs correction”, still does not seem to be working as intended because many of the interviewers checked off “No, cannot locate housing unit” rather than correcting the address for the housing unit.

9. For some match codes, Question A response "Cannot locate" is not aligned correctly. It is under the "No" responses, instead of one of the four main responses.

10. For Question L, the answer categories are out of traditional order. They are currently one, more than one, none. Need to discuss if there was a reason, and if not they should be in order -- none, one, more than one. When a unit is unfit, demolished, etc, the tendency is to want to mark "one" instead of "none" -- with none being the last on the list.

11. It was often unclear which BSA, Question L referred to when there were several addresses involved. The interviewer & QC interviewer interpreted it differently so the unit designations listed were not the same. For multiunit addresses - sometimes the interviewer would answer question L as "1" and sometimes it would be answered as "32", for the number of units at the BSA, which resulted in multiple "failings" during the QC process. One recommendation is to fill in the "BSA,” rather than rely on the lister's memory of the definition of a "BSA.” Also, the lister is reading this question aloud to the resident who doesn't know what we mean by "BSA" - they're going to provide the answer that makes sense to them, such as "there's 1 housing unit here". Since substituting the actual address rather than using the word "BSA" would clear this up, we should pre-fill with actual address.

12. Need to revisit placing addresses within the text of the question, having Census Address or CCM Address at the end of the questions caused some confusion.

13. On the M/DE form, (matches and possible Census duplicates), the break between the M and the DE section needs a banner similar to other forms. Also, need to discuss why we are collecting a map spot when we have established duplication.

14. “Non-linearity” of the questions was not well understood. It is recommend that:

(1) the questions be re-ordered so that respondent questions are grouped together and those requiring detective work be together; and (2) ALL respondent questions be printed in bold to distinguish them from the questions not intended for respondents. The form did not have a consistent use of BOLD to indicate a question for the respondent.

15. The space between lines in the "notes" area is too small to write in, may need to add more space between lines. In 2000, the interviewers were instructed to write on the back of the form if more space was needed. Need to discuss best approach for 2010.

16. "Do the following addresses refer to the same housing unit" can be confusing in certain circumstances. Often times the question is, "does ABC apartment 3 refer to the same housing unit as ABC apartment 3,” that is, the same address. When the lister reads that to themselves and/or the resident, it's an obvious yes. BUT in looking up at the info at the top of the page, you're really asking "does CCM map spot 6 match Census map spot 8". That is, are these similar addresses really located in the same place—should the map spots be in the same area? And if that is the question, need to ask differently. Otherwise, the interviewer doesn't know why they're asking the question and they're liable to come back without the needed information.

17. Special questions weren’t always questions, because analysts needed additional info not found on the follow-up form. Some questions may have been redundant, because the analysts are trying to draw attention to the FLD signifying something is wrong with an address. Need to discuss further with Matching Team.

18. For CCM map spots, should also include WMSNs on the form.

19. It was awkward to ask questions about an address that was not the correct address when there was more than one match code on a form, for example NE/DE, and the interviewer already got/verified the address for the first follow-up unit. The questions about the follow-up unit were already asked.

20. There was confusion regarding question A...is there a housing unit at <address> regarding boarded up units. In the definition of HUs, one example given was "a boarded up housing unit as long as it is not open to the elements (vacant or occupied). However, boarded up is listed under the no responses. It should be listed under the yes responses.

21. It was observed several times that a unit did not exist and the interviewer checked “other” rather than “cannot locate”, causing them to answer additional questions that did not make sense to answer. If they had checked instead the, “cannot locate” alternative, they would have skipped the additional questions. Need to discuss if a “does not exist” option should be a choice for question A1 along with the “cannot locate” option, or if training should better explain the difference between the “other” and “cannot locate” responses. “Does not exist” was a choice in 2000, but analysts wanted to remove it so interviewers would explain why it does not exist.

22. It was observed several times for the P (possible match cases) form where the interviewer answered “NO” to Question C and preceded to answer questions for the CCM and Census address. One of the addresses was incorrectly written down (that is, it didn’t exist), so it did not make much sense to answer the rest of the questions for that particular address (offshoot from comment above #13). Question C needs to be reworded or explained more in training to cover the confusion.

23. The IHUFU form left too much room (and necessity) for rewording the questions. If rewording or probing wasn’t done, the respondent was confused and/or a complete response was not provided. This is fine for interviewers who really understand the purpose of IHUFU and the purpose of each question, but risky for those interviewers that do not have a thorough understanding.

24. Multiunits- The IHUFU questions seemed more directed toward the BSA than the housing unit even though the housing unit seemed to be in question. In some situations it was hard to determine if the response is for the BSA or an individual housing unit. Need to discuss if dealing with apartments, we should add another check box to indicate, “the BSA was correct, but there was no apartment with that number.”

25. Need to discuss for the future construction response category, if the reasons from listing should be included.

26. When the question asks if there are any other listings in the reference sheet for CCM or Census, it was not clear to all what was meant by other than the ones already listed at the top. A lot of time was wasted filling in the numbers listed at the top of the sheet, when only supposed to be checking for other addresses in the reference list.

27. On the GQ Form, Question H could be interpreted as a negative, evaluative statement about the GQ. Suggest replacing "type of place this is" with "this place or address.”

28. In general, the definition of a GQ was a point of contention between a few participants and needs further discussion. A few examples are:

• College run apartments

• Assisted living apartments without 24 hour skilled nursing

According to the definitions, the first bullet is a GQ and the second is NOT, but the word apartment throws it off.

D. Initial Housing Unit Followup Quality Control Form:

1. Need to review the critical errors for QC and the acceptance table, as two interviewers can interpret some questions, such as C, differently but in the end the same information over a housing unit is gained. It was observed during QC that more than half of the time a cluster failed because of this, and had to undergo rectification, but probably should not have failed.

2. Regarding the identification of critical errors, the table in the manual appendix is difficult to understand and use. Also, we know that interviewers don’t usually carry their manuals with them. The annotated questionnaire alternative that was created to serve the same purpose is better, but still may be confusing and is, visually, very “busy.” Since nearly all of the errors are critical, could the non-critical errors be printed on the DQC form with an instruction that all other errors are critical?

3. Can page numbers for each case being followed up in QC be added to the QC form to help the interviewer find them fast?

4. Need to revisit why the number of cases rectified is counted. Shouldn’t it total the total HUs followed up?

5. Need to discuss adding a box in the bottom right corner of each form for QC to check ok (completed), or should it be Pass/Fail?

6. The QC check needs a different introduction than that listed on the IHUFU cover page. The following was used during the Mini IHUFU Test:

Hello. I am (your name) from the U.S. Census Bureau. Here is my identification. Over the last few days Census employees were checking addresses in this area as part of the 2008 Census Dress Rehearsal. I am here to check the quality of their work. My questions should only take a few minutes. This notice explains that your answers are confidential. (Hand the respondent a Privacy Act notice and allow time for him or her to read it.)

7. Eliminate Section E: Acceptance Table and only print out the allowable number of errors for the cluster's sample size, i.e., total number of cases selected for the QC Check. A recommendation is to pre-print an explicit instruction on the DQC form stating instructions such as: “Pass = 3 or fewer critical errors, Fail = 4 or more critical errors.”

8. Need to discuss Section D: Rectification. Instead of listing the individual blocks, if there is a QC box on each form print out only the total number of follow-up cases for the cluster and have the QC Checker enter the total number of incorrect cases, i.e., case forms with one or more critical errors.

E. Flashcard

1. Need the definition of future construction on the flashcard.

2. Distinguish the GQs and HUs on the flashcard, so interviewers do not need to determine in the field.

5. Maps

This section compiles the lessons learned and suggested improvements for the IHUFU Maps.

1. It was very hard to locate Census MSNs, since they are not sequential. Unfortunately, CCM has no control on the Census map spot numbering.

2. The Census MSNs are small and hard to find. Each interviewer needs a magnifying glass (or access to one) in order to locate the Census MSNs. Need to discuss the scale of map spots with Geography Division for 2010.

3. Label the maps better. Since the Census maps were received late, the labels on maps were not clear. We could not tell what was a CCM and what was a Census Map. This was a problem specific to the mini-IHUFU test.

4. Need to discuss if blank sketch map pages (blank pages with block cluster geographic information) should be provided to interviewers in their kits, similar to IL, so they can draw if they need to (might also encourage it as a form of notes).

5. A recommendation was made to make two copies of the CCM block map; a clean map for use in spotting Census cases missed in CCM for future CCM Operations and a map for the field interviewers to mark-up and highlight as they see fit to complete the cluster. If the Census MSN is present, the interviewer could highlight it on their copy of the map, and insert it in the approximate location on the CCM map where it belongs. Need to discuss, as it may cause problems if map spots are not put on the map for future CCM operations, and only the other copy.

6. Need to discuss if maps should be corrected during QC?

7. Some of the CCM Block Cluster maps were unclear, for example could not distinguish an MSN of eight from a nine. This may have been specific to the mini-IHUFU using scanned images, but for 2010, need to add a procedure to the Independent Listing Office Manual to have the maps checked for quality/readability after they are photocopied to be stored for use in future CCM operations.

8. Specific to the mini-IHUFU test, the surrounding block cluster maps were difficult to use. In 2010, block cluster maps will be labeled with the surrounding blocks on it to identify their relation to the cluster.