NASS Comments

March 11, 2016

From: Brian Richards, Statistician in the Summary, Estimation and Disclosure Methodology Branch in the Methods Division of NASS

Subject: My Review of Proposed Data Collection: Community Partner Interview (CPI) Survey for SNAP / FNS

I carried out an assessment of this proposed survey, based on the documents contained in the package provided to me. FNS (Food and Nutrition Service) is attempting to gain OMB approval to collect information from SNAP (Supplemental Nutrition Assistance Program) participants. Staff at Community Based Organizations (CBOs) are serving as representatives of the SNAP state agencies during the SNAP interviews. FNS will use the information from this study to assess the impact of CBOs on the SNAP interview process.

An important concern / question that I have relates to a statement in the “Part B” (Statistical Methodology) document. On page 1 of this document is the statement “Convenience sampling will be used to create the sample”. However, there is no elaboration or explanation of what the authors mean by convenience sampling. For illustration purposes, let us assume that a given SNAP office is open Monday through Friday, from 9am to 5pm. Will the sample of interviews that will be conducted (by CBOs) at times that are convenient for them, for example? Maybe they will only conduct there interviews between 1pm and 4pm, for example? If so, the sample may not be representative of all potential SNAP participants, since maybe certain types of clients are more likely to visit these offices in the morning? Or maybe the interviews will only be conducted on Wednesdays and Thursdays? Perhaps these days are more “convenient” for the CBO staff involved. But, again, the sample may not be representative of all potential SNAP participants, since maybe certain types of clients are more likely to visit these offices on a Monday or a Friday? The sample design should be set up such that persons are able to be sampled across all times that the SNAP offices are open for business.

Another key question in my mind is where these interviews will be conducted? Different references in the Part A and Part B documents seem to give contradictory information. In the Part A document (bottom of page 1, top of page 2), there is a statement “ … collect information from participating SNAP offices and a sample of SNAP applicants who were interviewed by CBO staff persons when applying for benefits”. So this statement seems to say that the interviews will be conducted in the SNAP offices. But in the Part B document (top of page 2), there is a statement “ … the respondent will be instructed … in each CBO office”. So this statement seems to say that the interviews will be conducted in CBO offices. In the Part B narrative, beginning on page 1, the authors introduce the acronym CPI without defining it. I assume CPI stands for community partner interview, given the title page of this document. Related to this, near the bottom of page 1, there is a statement “Determine the factors … at a CPI instead of a SNAP office”. This seems to infer that the CPI is a physical location / office. Since the last part of the sentence refers to “instead of a SNAP office”, perhaps the CPI (in this context) is a location where a CBO is housed?

Perhaps some of the interviews would be conducted at SNAP offices and some at CBO offices … and the authors just have not explained this? If my hypothesis is correct, it would seem like part of the assessment would involve looking for “differences” between interviews conducted at each place.

In the part B document there is a reference to confidence intervals associated with customer satisfaction ratings (where the ratings were in terms of a percentage, as I understand it). What I did not see in any of the explanations was reference to a comparison of these types of percentages back to such ratings associated with SNAP staff conducting the interview with the SNAP applicants. For example, perhaps the customer satisfaction (of SNAP applicants) ratings for interviews conducted by SNAP staff is 86%, and the customer satisfaction ratings for interviews conducted by CBO staff is 84%. Based on the statistical test carried out, this difference may not be statistically significant. But if these percentages were 89% and 82%, for example, perhaps this difference would be significant. As I understand the basis for conducting this study / data collection, these types of comparisons would seem to be applicable.

My next set of comments relates to the questionnaire that is planned to be used. From the documents that were sent to me, I see two different versions of the questionnaire. I am not sure which is the final version of the questionnaire that will be used? The first attachment (in the email message that our branch received) was Appendix C, which is an English version of the questionnaire. There are 6 multiple choice questions and the questions seem reasonable to me. Attachment 7 is a spreadsheet, and the 6^th tab (apparently) also represents the questionnaire. But this questionnaire version displays 10 questions. Perhaps one set of questions was an early draft and the other set of questions was a final version?

My final set of comments are less technical in nature. In the Part A document, around the middle of page 5 is the statement “FNS consulted with xxx at the NASS …”. I assume that my name should replace the “xxx”, and Edwin’s name should be replaced by David Hancock, given the job title referenced. There is a similar statement in the Part B document. The very last sentence on page 6 references Edwin’s name. I assume that this should be replaced by my name.