NASS Review Document

October 21, 2008

Dave Dillard

This document analyzes the Forest Service’s OMB request to conduct a phone (CATI) survey to collect information about media sources certain race/ethnic groups (White, Hispanic, Asian, and African American) use to gather information. Once the Forest Service identifies specific media sources used by these groups, it plans to use these media to acquaint the groups with information on recreational uses of national forests located close to urban areas.

The request provided ample documentation of differences among the four ethnic groups of interest with respect to how they obtain information. The documentation was also well supported by subject matter experts contacted by the Forest Service for advice on how to proceed with the survey.

Since this survey is not a probability survey, I believe the steps outlined to identify potential respondents are sound. They are obviously well thought out and, like other documentation in the request, well supported. One of the subject matter experts, Dr. William Borrie, discussed weaknesses to phone surveys and suggested more extensive use of Internet surveys. I cannot disagree with Dr. Borrie’s statements. However, for this survey, I believe phone data collection is adequate and will provide the information the Forest Service desires.

The Forest Service request states that it expects an 80 percent response, but doesn’t say how it arrived at that estimate. It also plans to make up to 12 follow-up calls to non-answers, which seems ample. The Forest Service said it had pre-tested the questionnaire on nine individuals, collecting data in person and asking respondents about possible confusion with the questions. I applaud the Forest Service for asking respondents to identify wording problems in the instrument, but I suggest that it conduct more pre-testing using operational (i.e., phone) procedures.

Overall, the Forest Service request is well documented, well supported by research results and advice from subject matter experts, and consistent with the legislation mentioned in PL-95-307 and EO 12898.

My final comments concern the questionnaire (CATI instrument) and data collection procedures. When I collected survey data, I avoided offering potential respondents a chance to discontinue the interview. (However, we are required by law to tell the respondent that cooperation is voluntary.) I’ve noticed that phone solicitors have become more aggressive in their techniques. Instead of asking me whether I would like to contribute, they ask if I’m comfortable giving a specific amount which they designate. In keeping with that philosophy, I would drop the introductory question “Would you like to participate?” At the least, I would re-phrase it as “Would you be willing to participate?” I would also drop the statement “I promise I’m not trying to sell you anything,” which could arouse suspicion. If the Forest Service is interested in collecting data from 18-year-olds, it should replace the screening question “Are you over 18 years of age?” with “Are you 18 years of age or over?” Instead of asking “Would you mind telling me your ethnicity, please?” I would ask “What is your ethnicity?” or “Which ethnic group do you most closely identify with?” Question 6 asks respondents which media source they would trust the most to obtain information about outdoor recreational opportunities. I wonder whether “trust” is the best word to use in that question. Finally, the skip instruction in Question 7 could cause confusion. In a CATI instrument, the interview should automatically branch to the correct continuation based on the response to the question. However, there should be a branch after Question 20 so the interviewer doesn’t ask Question 21, which is identical to Question 19. Finally, the documentation alluded to obtaining income data for categorization purposes, but no income questions appear in the draft questionnaire.

2