ATTACHMENT C -NASS Comments and Responses

ATTACHMENT C:

NATIONAL AGRICULTURAL STATISTICS SERVICE (NASS) COMMENTS AND RESEARCH TEAM RESPONSES

FDPIR OMB Package Review

A Study of Food Distribution Program on Indian Reservations

1. Comments received from NASS, January 2013

Overall

The documentation provided for review was thorough and well written. The survey methodology is sound and seems to address the defined study objectives.

Interview Length

There is some inconsistency throughout the documentation in regards to the expected time to complete an interview of participants. The time is reported at 30, 40, and 45 minutes

Household Interview Sample

The documentation does not adequately detail how the 998 households will be selected to be interviewed. Part A details about 32 records from each of the 25 sites will be selected. Part B has 32 records per site, excluding the two largest sites which will have 68 each. The total sample described in Part B would result in 872 samples, 126 samples less than Part A reported.

Discussion Group

Discussion group selection may be biased by convenience sample methodology. Certain FDPIR or Tribal program staff could influence the discussion by only recruiting participants of a one strong opinion (i.e. only members in strong favor of the program are recruited). The number of participants per discussion group is not consistent between the test group and the final proposal. After reading the documentation, it was unclear the intended number of participants per group.

Specific Document Comments

Part A

• Introduction

  • “including the elderly, living on Indian reservations,..” is difficult to follow, are only the elderly eligible? A potential rewording “including the elderly, individuals living on Indian reservations,…”

• Case Record Reviews

  • The criteria for selecting samples is not detailed, however is described in Part B. It may be beneficial to note the location of this documentation in Part A.

• A11, The survey instruments ask participants about health related issues and illnesses; would these items be considered to be of a sensitive nature?

Part B

• Page 12, details that participation change (2006-2011) will be used to base the sampling of the 17 sites, however Attachment F describes 2001-2006.

• Pretest of Discussion Group Guide number of participants in each group does not coincide with counts Attachments C and L. Was it determined that a smaller group would be as effective as the larger group?

Attachment A

• Are the medical insurance (1.4) and health issues (1.11) questions relevant to this study’s objectives?

• How are current participants going to help explain why other participants have discontinued their participation? (4.6)

Attachment B1

• HOH is not defined prior to use.

Attachment B1A

• Students may be receiving multiple types of aid at the same time (i.e. Pell Grant and Student Loan). The form appears to allow for one type of student aid per person.

• How will joint savings and checking accounts be handled? Will the funds be divided amongst the individuals or will the funds be attributed to one individual?

Attachment B2

• Participant Informed Consent

  • Is the monetary or gift card thank you available to participants that end the survey early or only to participants that complete the survey?

  • Paragraph 3, the last two sentences required two readings to ensure that the report to the tribe/native village would not disclose individual data. A higher response rate may be achieved if these two sentences could be combined or reworded.

• Informed Consent For Use of Proxy

  • See comment on Attachment B2, Participant Informed Consent, paragraph 3

• Household Enumeration

  • Question 5 regarding marital status

    • Is the intent to obtain the number of legal marriages? Some couples may refer to their spouse or marriage without a legal union. The question is restricted to those 18 and older, there may be some household members under 18 that are legally married.

  • Question 8 regarding employment

    • If household income earned by those under 18 is included in eligibility, then it may be relevant to collect that data as well.

  • Section B

    • Question B1a is vague and could be interpreted many different ways by the respondent. For example the food didn’t last because it was consumed or the food didn’t last because it spoiled.

    • Question B1c, B1d, and B1e appear to be asking the same thing. The questions need to be modified to make differences clearer.

    • Question B5 been is missing from the question, i.e. “in your household been referred to”

    • Question B7, the question is ambiguous. Is the intent to determine the proportion of meals prepared at home or to determine the proportion of meals prepared in the community group?

  • Section D

    • In many areas, question D1a and D1e are referring to the same stores (i.e. Target and Walmart).

    • Question D1d and D1e Walmart and Target are both considered general or discount stores. Perhaps question D1d could refer to Sam’s Club or Costco?

Attachment B4

• Module D

  • Question D7 missing to in the question: “…use compters to screen households…”

• Module G

  • Looking at Walmart’s current website, Walmart is no longer hyphenated. See comments on Attachment B2 regarding classification of Walmart and Target.

  • Is question G16 asking for clarification of question G15?

Attachment F

• The growth code is based on data from 2001 to 2006, this change is from 6 years in the past. To examine the current state of the program, the growth code is expected to reflect a more recent time frame, such as 2006 to 2011.

• The growth code for sites sampled with certainty should be included in the documentation. This would help show if the increase or decrease in participation for the larger sites. Currently 6 of the 19 non-certainty sampled sites show a decline, was the growth code used in selecting samples?

Attachment H2 and H3

• The increase in sample size on the two largest tribes is almost double, which exceeds “a little more”.

Attachment G

• Definitions of NAFDPIR and IRB are not provided prior to the first use of the term.

Attachment H5

• On-site visits are listed as 17 of the 26 sites, previous portions of the survey refer to 17 of 25 sites.

Attachment L

• Other portions of the survey documentation mention 20 discussion groups, however this document states 19.

FDPIR OMB Package Review

A Study of Food Distribution Program on Indian Reservations

2. Responses to comments received from NASS, January 2013

Overall:

Interview Length: There is some inconsistency throughout the documentation in regards to the expected time to complete an interview of participants. The time is reported at 30, 40, and 45 minutes

This comment has been addressed throughout the document (Parts A and B) to assure consistency.

Household Interview Sample: The documentation does not adequately detail how the 998 households will be selected to be interviewed. Part A details about 32 records from each of the 25 sites will be selected. Part B has 32 records per site, excluding the two largest sites which will have 68 each. The total sample described in Part B would result in 872 samples, 126 samples less than Part A reported.

These numbers have been corrected and checked for consistency throughout the document (Parts A and B).

Discussion Group: Discussion group selection may be biased by convenience sample methodology. Certain FDPIR or Tribal program staff could influence the discussion by only recruiting participants of a one strong opinion (i.e. only members in strong favor of the program are recruited).

Reasons why fewer individuals are participating in FDPIR and participant satisfaction will be addressed primarily through interviews and case file reviews. The discussion groups will add important information to those data collection components, providing more nuanced information about the choices people make regarding FDPIR participation. Discussion group participants will not constitute a representative sample of FDPIR participants or eligible nonparticipants, and the discussion group findings will not be generalizable to the larger FDPIR participant and eligible nonparticipant population. Nevertheless, every effort will be made to minimize bias in recruiting participants. This will be accomplished through: 1) clear guidelines to Tribes regarding the purpose of the discussion groups and the identification and recruitment of participants; 2) discussions with Tribes during outreach and site visit scheduling to determine the best recruitment strategies for each Tribe; and 3) project staff training to facilitate working with Tribes to guide and assist in the recruitment process and related logistics involved in arranging the discussion groups. With the assistance of FDPIR managers and staff, current and former FDPIR participants can be identified through case files; project staff will reach out to other Tribal programs, such as WIC, programs serving elders, and food banks that serve reservation residents, to recruit non-FDPIR low-income households.

The number of participants per discussion group is not consistent between the test group and the final proposal. After reading the documentation, it was unclear the intended number of participants per group.

The pretest was not intended to simulate the discussion groups as planned for the study. Doing so would have been too costly and imposed an additional burden on FDPIR staff and discussion group participants. The pretest for the discussion groups was limited to testing the questions in the guide with two individuals, one an FDPIR participant, and one a low-income individual not participating in FDPIR.

For the study, OMB approval is requested for a total of 19 discussion groups—one in each of 15 sites and two groups in each of the two largest sites. Each group is planned to consist of 12 adults. To ensure that the desired number of people attends the focus group sessions, we will need to over-recruit to allow for the incidence of no-shows. We estimate that we will need 16 people to agree to attend a focus group session to ensure a group of sufficient size.

Specific Document Comments

Part A:

Introduction: “including the elderly, living on Indian reservations,..” is difficult to follow, are only the elderly eligible? A potential rewording “including the elderly, individuals living on Indian reservations,…”

Since elderly are included in the low income populations living on Indian reservations that are served by FDPIR, the phrase was edited to read: “FDPIR provides USDA Foods to low-income households living on Indian reservations…”

Case Record Reviews: The criteria for selecting samples is not detailed, however is described in Part B. It may be beneficial to note the location of this documentation in Part A.

The following statement has been added to Part A, page 3: “Details about the selection of the sample are included in Supporting Statement Part B, response 1b.”

A11: The survey instruments ask participants about health related issues and illnesses; would these items be considered to be of a sensitive nature?

Yes, they would be considered sensitive. This has been added to A.11 along with justification for asking about these topics.

Part B

Page 12, details that participation change (2006-2011) will be used to base the sampling of the 17 sites, however Attachment F describes 2001-2006.

For the selection of the sample, we looked at participation trends from 2001-2011, sorting on whether participation over this period fell by more than 25 percent, fell by less than 25 percent, or increased. The dates on Attachment F have been corrected accordingly.  For selecting the purposive sample of 17 sites, we will consider, among other factors, participation change for the same period, 2001-2011, ideally including sites with increasing as well as decreasing participation. We have made the correction in Supporting Statement Part B.

Pretest of Discussion Group Guide number of participants in each group does not coincide with counts Attachments C and L. Was it determined that a smaller group would be as effective as the larger group?

For the study, OMB approval is requested for a total of 19 discussion groups—one in each of 15 sites and two groups in each of the two largest sites. Each group is planned to consist of 12 adults. The numbers in Attachment L have been corrected.

Attachment A:

Are the medical insurance (1.4) and health issues (1.11) questions relevant to this study’s objectives?

These question address Study Objective 1, Provide a demographic profile of households and individuals that currently participate in the FDPIR, including their participation in other assistance programs. FDPIR is concerned about providing foods that are compatible with the nutrition and health needs of the eligible population and, for that reason, health issues related to dietary restrictions or requirements are relevant. FDPIR is also interested in promoting health and nutrition education, and questions about health insurance indicate sources of care which are important opportunities for coordination of nutrition education for FDPIR participants

How are current participants going to help explain why other participants have discontinued their participation? (4.6)

FDPIR participants do go on and off of the program during the course of a year, so current FDPIR participants may in fact have discontinued FDPIR in the past year. Some reasons for this (e.g., changes in eligibility) can be obtained from case records, while other reasons (individual preferences, program satisfaction) will be obtained during interviews.

Attachment B1:

HOH is not defined prior to use.

A definition of Head of Household (HOH) has been added to Attachment B1a.

Students may be receiving multiple types of aid at the same time (i.e. Pell Grant and Student Loan). The form appears to allow for one type of student aid per person.

Attachment, B1b, the case record review form, has been revised to indicate “Mark all that apply” for this item, and will allow for multiple selections.

How will joint savings and checking accounts be handled? Will the funds be divided amongst the individuals or will the funds be attributed to one individual?

This information is being pulled from the case files.  The data will replicate the way it is entered in the case file.

Attachment B2:

Participant Informed Consent:

• Is the monetary or gift card thank you available to participants that end the survey early or only to participants that complete the survey?

The gift card will be available to participants that end the survey early. This has been clarified in the informed consent.

• Paragraph 3: the last two sentences required two readings to ensure that the report to the tribe/native village would not disclose individual data. A higher response rate may be achieved if these two sentences could be combined or reworded.

The end of paragraph 3 has been reworded as follows: “A report that summarizes the survey findings will be shared with the tribe/native village at a later date, but it will not include names of individuals or tribes/native villages.”

Informed Consent For Use of Proxy: See comment on Attachment B2, Participant Informed Consent, paragraph 3

Informed consent for use of proxy has been edited consistent with the edit in participant informed consent: “A report that summarizes the survey findings will be shared with the tribe/native village at a later date, but it will not include names of individuals or tribes/native villages.”

Household Enumeration:

Question 5 regarding marital status: Is the intent to obtain the number of legal marriages? Some couples may refer to their spouse or marriage without a legal union.

No, legality is not important—in the household enumeration, we want the information to reflect how the household members view their relationships. We have slightly revised the question to read “current marital status.”

The question is restricted to those 18 and older, there may be some household members under 18 that are legally married.

Again, legality is not an issue.  Since we do have SNAP cases composed of 17 and 18 year olds without an adult, we have changed the restriction on this question to 16 and older rather than 18 and older.

Question 8 regarding employment: If household income earned by those under 18 is included in eligibility, then it may be relevant to collect that data as well.

We will exclude income earned by those under 18. The FDPIR Handbook provides the following guidance: The earned income of members of the household who are students at least halftime, and who are not yet 18 years old is excluded even during temporary interruptions in school attendance due to semester or vacation breaks, provided the child's enrollment will resume following the break.  Individuals are considered children for this exclusion if they are under the parental control of another household member. 

 Section B:

Question B1a is vague and could be interpreted many different ways by the respondent. For example the food didn’t last because it was consumed or the food didn’t last because it spoiled.

This is a validated measure and cannot be changed. Source: US Household Food Security Survey Module: Six-Item Short Form [HH3, HH4, AD1, AD1a, AD2, AD3]

Question B1c, B1d, and B1e appear to be asking the same thing. The questions need to be modified to make differences clearer.

Same reason as above. The questions address three distinct states of potential food insecurity: cut the size of your meals or skip, eat less than you should, ever hungry. We have put these phrases in italics so that they stand out.

Question B5 been is missing from the question, i.e. “in your household been referred to”

This question has been edited to read as follows: “Has anyone in your household been referred to other food programs by the FDPIR staff?”

Question B7, the question is ambiguous. Is the intent to determine the proportion of meals prepared at home or to determine the proportion of meals prepared in the community group?

We are interested in meals prepared outside the home. The question had been reworded as follows: “During the past month did you or anyone in your household get any meals that were prepared outside your home by extended family or prepared by a community group?”

IF YES: How often did the household eat meals prepared outside your home by extended family or prepared by a community group? Would you say none, some, about half or most meals?

Section D:

In many areas, question D1a and D1e are referring to the same stores (i.e. Target and Walmart). Question D1d and D1e Walmart and Target are both considered general or discount stores. Perhaps question D1d could refer to Sam’s Club or Costco?

Question D1d has been reworded as follows: “ Nearest warehouse/ department/large, big-box store that sells groceries (such as Walmart, Target, Costco, Sam’s Club).”

Attachment B4

Module D: Question D7 missing to in the question: “…use compters to screen households…”

This correction had been made.

Module G:

Looking at Walmart’s current website, Walmart is no longer hyphenated. See comments on Attachment B2 regarding classification of Walmart and Target.

This correction has been made.

Is question G16 asking for clarification of question G15?

Yes, format and numbering has been changed to reflect this.

Attachment F:

The growth code is based on data from 2001 to 2006, this change is from 6 years in the past. To examine the current state of the program, the growth code is expected to reflect a more recent time frame, such as 2006 to 2011.

As noted in our response to the question on Supporting Statement Part B, the dates on Attachment F have been corrected to reflect that we looked at participation trends from 2001-2011.

The growth code for sites sampled with certainty should be included in the documentation. This would help show if the increase or decrease in participation for the larger sites. Currently 6 of the 19 non-certainty sampled sites show a decline, was the growth code used in selecting samples?

Growth codes for the certainty sites have been added to Attachment F. Both samples (certainty and noncertainty) were selected with probability proportionate to size.  Then, we stratified, sorting the programs first on region and then on whether participation between 2001 and 2011 fell by more than 25 percent, fell by less than 25 percent, or increased. 

Attachment H2 and H3:

The increase in sample size on the two largest tribes is almost double, which exceeds “a little more”.

The wording in these letters has been changed as follows: “About 32 FDPIR household records will be reviewed at the FDPIR office for each Tribe, with about twice as many at the two largest Tribes.”

Attachment G: Definitions of NAFDPIR and IRB are not provided prior to the first use of the term.

This has been corrected.

Attachment H5: On-site visits are listed as 17 of the 26 sites, previous portions of the survey refer to 17 of 25 sites.

This has been corrected to read 17 of 25 sites.

Attachment L: Other portions of the survey documentation mention 20 discussion groups, however this document states 19.

There will be 19 discussion groups, one in each site except for the two largest sites, each of which will have two discussion groups. This has been corrected in Attachment L.