50305 DCM FRP_OMB PART B Final

50305 DCM FRP_OMB PART B Final.docx

Evaluation of the Direct Certification with Medicaid for Free and Reduced-Price (DCM-F/RP) Meals Demonstration

OMB: 0584-0630

Document [docx]
Download: docx | pdf


SUPPORTING STATEMENT - PART B for

OMB Control Number 0584-NEW:

Evaluation of the Direct Certification with Medicaid for
Free and Reduced-Priced (DCM-F/RP) Meals Demonstrations









Conor McGovern

Research Analyst

COR/OPS



USDA, Food and Nutrition Service

3101 Park Center Drive

Alexandria, Virginia 22302

CONTENTS

PART B. COLLECTION OF INFORMATION USING STATISTICAL METHODS 1

B.1. Respondent Universe and Sampling Methods 1

B.2. Procedures for the Collection of Information 9

B.3. Methods to Maximize Response Rates and Deal with Non-Response 16

B.4. Test of Procedures or Methods to Be Undertaken 18

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 19


APPENDIX A: QUALITATIVE DATA COLLECTION MATERIALS

A.1. SITE VISIT LETTER

A.2. SITE VISIT SCHEDULING CALL PROTOCOL

A-2.a. SITE VISIT PREPARATION: STATE CHILD NUTRITION AGENCY

A-2.b. SITE VISIT PREPARATION: STATE MEDICAID AGENCY

A-2.c. SITE VISIT PREPARATION: DISTRICT

A-2.d. VENDOR TELEPHONE INTERVIEW EMAIL

A-3.a. Detailed site visit interview protocol: state child nutrition agency staff

A-3.b. Detailed site visit interview protocol: state Medicaid agency staff

A-3.c. Detailed site visit interview protocol: district staff

A-4. Site visit observation protocol

A.5. Site visit follow up telephone interview email

A.6.a. Detailed Follow-up interview protocol: state child nutrition agency staff

A.6.b. Detailed Follow-up interview protocol: state medicaid agency staff

A.6.c. Detailed Follow-up interview protocol: district staff

APPENDIX B: ADMINISTRATIVE RECORDS REQUEST

B-1. ADMINISTRATIVE RECORDS REQUEST EMAILS

B.2. ADMINISTRATIVE RECORDS REQUEST

APPENDIX C: STATE COST DATA COLLECTION MATERIALS

C-1. STATE COST LOG EMAILS

C-2. INSTRUCTIONS FOR STATE COST DATA TRACKING LOGS

C.3.A. STATE CHILD NUTRITION AGENCY COST LOG

C.3.B. STATE MEDICAID AGENCY COST LOG

C.4. SEMI-STRUCTURED STATE COST LOG CLARIFICATION
APPENDIX D: CONFIDENTIALITY PLEDGES

APPENDIX E: ANNUAL BURDEN ESTIMATE

APPENDIX F: PUBLIC COMMENTS AND RESPONSES

F1a: PUBLIC COMMENT #1

F1b: FNS RESPONSE TO PUBLIC COMMENT #1

F2a: PUBLIC COMMENT #2

F2b: FNS RESPONSE TO PUBLIC COMMENT #2

F3a: PUBLIC COMMENT #3

F3b: FNS RESPONSE TO PUBLIC COMMENT #3


APPENDIX G: NASS COMMENTS AND RESPONSES

APPENDIX H: RELEVANT DOCUMENTS

H.1. Section 104 of the Child Nutrition and WIC Reauthorization Act of 2004

H.2. Section 103 of the Healthy, Hunger-Free Kids Act of 2010




TABLES

B.1.1 Respondent Universe and Sample Size 2

B.1.2 Districts Included in Qualitative Data Collection Subsample 6

B.1.3 Sample Sizes and Expected Response Rates, by Research Activity 9

B.3.1 Expected Response Rates and Methods to Maximize Response Rates, by Research Activity 17

B.5.1 Individuals Consulted 19




PART B. COLLECTION OF INFORMATION USING STATISTICAL METHODS

B.1. Respondent Universe and Sampling Methods

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

Beginning in SY 2016–2017, FNS initiated a new demonstration of Direct Certification with Medicaid for Free and Reduced-Price (DCM-F/RP) Meals, under the administrative pilot authority in Section 18(c) of the Richard B. Russell National School Lunch Act (NSLA). This demonstration permits the selected States to use data from Medicaid files to identify students eligible to receive free or reduced-price meals and directly certify them at that level. FNS seeks approval to conduct data collection as part of the Evaluation of the DCM-F/RP Demonstration.

The demonstration was awarded based on a competitive RFA process. The Cohort 1 (SY 2016-17) States were chosen in the first round of the competitive RFA process from the pool of States that applied. A subsequent competitive RFA process was used to select Cohort 2 (SY 2017-18) States. States were chosen based on applications in which they demonstrated that they could successfully comply with all requirements and carry out the demonstration project, including providing all necessary data for the required evaluation. Direct certification with Medicaid data is currently only carried out through demonstration projects, including the original DCM demonstrations and the current DCM-F/RP demonstrations, which will be evaluated in Year 2 with this information collection.

The respondent universe for the Evaluation of the DCM-F/RP Demonstration includes the 15 State Child Nutrition Agencies, 15 State Medicaid Agencies, and 3 other State Agencies involved in the DCM-F/RP process in the 15 States that are conducting DCM-F/RP; the 7,218 school districts within these 15 States; and 4 vendors involved in the DCM-F/RP process (Table B.1.).

Table B.1.1. Respondent Universe and Sample Size

Respondent type

Respondent universe

Sample size

State Child Nutrition Agenciesa

15

15

State Medicaid Agenciesa

15

15

Other State Agencies

3

3

School Districtsb

7,218

34c

Vendors

4

4

a Our burden estimates assume an average of three respondents within each Child Nutrition and Medicaid agency, but the sampling unit and unit of analysis is the agency (or the district), not the individual, and response rates will be computed at the agency level.

b The respondent universe for school districts was estimated using the number of districts in each sample state that submitted FNS-742 data for SY 2015-2016. Our burden estimates assume an average of three respondents within each district, but the sampling unit and unit of analysis is the district, not the individual, and response rates will be computed at the district level.

c Although qualitative data will be collected from a sample of 34 districts, all 7,218 districts in the demonstration will be included in the administrative records data provided by State Child Nutrition Agencies.


Summary of Prior Related Activity

The DCM-F/RP demonstration builds on an earlier demonstration of DCM for free meals (OMB Control Number 0584-0586 The Evaluation of Demonstrations of NSLP/SBP Direct Certification of Children Receiving Medicaid Benefits, discontinued 4/30/2016), which began in SY 2012–2013. Under the DCM demonstration, FNS authorized seven States to directly certify students for free meals using income data available through Medicaid. In some demonstration States, districts were randomly assigned to conduct DCM or to a control group; in others, the demonstration was implemented statewide. The evaluations of this demonstration examined the effects of DCM on participation, costs, and other outcomes. Data collection activities included administrative records, cost logs completed by State Child Nutrition and Medicaid Agency staff, a web survey on district costs, and telephone interviews on challenges encountered by States and districts. Findings from the evaluation were published by FNS in 2015 and 2016.1 The response rates were 100 percent on the collection of administrative records, cost logs, and telephone interviews; and 80 percent or higher for all rounds of the survey in each State.

Data collection for the DCM-F/RP demonstration began with a pretest in seven States in SY 2016–2017. A previous OMB submission (Generic OMB clearance number 0584-0606 – Pretest for Evaluation of Direct Certification with Medicaid Demonstrations; clearance date 12/19/2016) covered the pretest. The current package requests clearance for data collection activities in SY 2017–2018. The pretest year of data collection includes staff from State Child Nutrition Agencies, Medicaid Agencies, and 16 school districts in seven States.2 Similar to the proposed information collection, staff are expected to participate in multiple data collection activities, including site visits, follow-up interviews, and, for some, the completion of cost tracking logs and administrative records requests. The data collection is in progress, so response rates have not yet been computed. In addition, pretest respondents will participate in debriefings to provide input into improvements to the data collection instruments and processes.

Selecting States

The sample for the DCM-F/RP evaluation will include 15 States.3 FNS solicited applications from States to participate in DCM-F/RP, and purposively selected these States from among those that applied to begin conducting DCM-F/RP. Therefore, the sample will include all States participating in the demonstration but will not be representative of any larger set of States.

Different components of the DCM-F/RP study address different research questions, and the data collection activities and samples of respondents differ. We will collect four key types of data:

  1. On-site interviews and observations (Appendices A-3a through A-3c and A-4);

  2. Site visit follow-up telephone interviews (Appendices A-6a through A-6c);

  3. Administrative data on certification and participation (Appendix B-2); and

  4. State-level cost data (Appendices C-3a and C-3b).

Identifying School Districts for Site Visits and Follow-Up Telephone Interviews

Some analyses—those based on administrative records from the States—will include all districts in each State. Other analyses—those requiring data collection from districts—will focus on a subsample of districts. The subsample will include 2 districts in each State that conducts direct certification matching centrally at the State level, with the exception of California.4 Because California will transition from operating the demonstration in a small subset of districts (14) in SY 2016–2017 to the entire State in SY 2017–2018, we will collect qualitative data from 4 districts in that State—2 that began in SY 2016–2017 and 2 that will begin the following year. We will also include 4 districts in the State that does not conduct central matching. The total number of districts in the subsample used for the qualitative analyses will be 34 (Table B.1.2). This district subsample will be selected purposively and will not be representative of any broader set of districts. We will use information from FNS-742 School Food Authority (SFA) Verification Collection Report (approved under OMB # 0584-0594 Food Program Reporting System, expiration date September 30, 2019) and the USDA Economic Research Service Urban-Rural Continuum Codes to assess diversity along key characteristics, including public and private schools, and ranging in size, urbanicity, and levels of F/RP meal certification. We will also solicit input from State Child Nutrition Agency staff on other characteristics, such as local data systems and procedures. FNS will ask the State Agencies to reach out to the districts to explain the study requirements and encourage them to participate.

Table B.1.2. Districts Included in Qualitative Data Collection Subsample


Number of
states

Number of districts
per state

Total number
of districts

California

1

4

4

Virginia

1

4

4

Other DCM-F/RP States

13

2

26

Total

15

N/A

34

Note: This assumes that the seven States that began conducting DCM-F/RP in SY 2016–2017 continue their current matching process (in which all but Virginia conduct matching centrally) and that all eight States beginning the demonstration in SY 2017–2018 conduct matching centrally that year.


Next, we provide specific information on the respondent universe for each of the four study components.

1. Sample for the site visit activities.

1. Sample for the site visit activities. We will conduct site visits in each of the 15 States to learn about the demonstration implementation process. During the visits, we will interview the staff at State Agencies, selected districts and vendors that play key roles in the DCM-F/RP process. The sample for these on-site interviews will include 45 State Child Nutrition Agency staff (3 per State), 45 State Medicaid Agency staff (3 per State), 3 staff from other State Agencies that play key roles in the DCM-F/RP process (1 in each of 3 states), 102 district staff (3 per district), and 4 staff from State Child Nutrition and Medicaid Agencies vendors that play key roles in the direct certification process (1 for each of 4 vendors). The interview participants will be the staff most knowledgeable about the demonstration, as identified by the main contact at each State agency or district.

The site visits will also include observations of key activities in the process. The sample for these on-site observations will be a subsample of the staff involved in interviews, including 30 State Child Nutrition Agency staff (2 per State), 30 Medicaid Agency staff (2 per State), and 64 district staff (2 per district). The observations will be led by the staff responsible for completing that step of the process.

Based on prior experience with similar studies and respondent universes, we expect to achieve 100 percent response from these State, school district, and vendor staff. To maximize response rates and ensure the highest quality data possible, State Child Nutrition Agency staff will contact districts selected for site visits. Senior members of the evaluation team will serve as the site visit leads and will work with our main State contacts to ensure that the relevant staff at each State Child Nutrition and Medicaid Agency are included in the visits and encouraged to participate.

2. Sample for the follow-up telephone interviews. We will conduct telephone interviews with respondents at the State Child Nutrition and Medicaid Agencies and districts included in site visits. The sample for these telephone interviews includes one staff person from each agency or district visited, for a total of 15 State Child Nutrition Agency staff, 15 State Medicaid Agency staff, and 34 district staff. Based on prior experience with similar studies, we expect to achieve 100 percent response from these State Child Nutrition Agency staff, 93 percent from State Medicaid Agency staff, and 91 percent from school district staff. We will contact respondents who are slow in responding to interview requests to answer their questions and remind them of the importance of the demonstration evaluation. If necessary, State Agency or FNS staff may contact any reluctant respondents to underscore the importance of study participation.

3. Sample for the administrative data on certification and participation. We will collect from State Child Nutrition Agencies district-level administrative data on certification and participation (meals served) for each district in the State. Where available, we will also gather information on DCM-F/RP match results and the certification status, method, and basis for matched students before that match. We will also collect statewide administrative data from each State on each district’s total number of reimbursable lunches and breakfasts served, by reimbursement category. The sample for this administrative data request will include 15 State Child Nutrition Agency staff. Based on prior experience with similar studies, we expect to achieve 100 percent response from these States. To ensure this high response rate, we will track respondents who have provided administrative records, and send email reminders to those who have not. We will contact respondents who are slow in responding to administrative records requests to answer their questions and remind them of the importance of the demonstration evaluation. If necessary, FNS staff may contact any reluctant respondents to underscore the requirements of study participation.

4. Samples for the State-level cost data. We will collect data on administrative costs incurred by staff of State Child Nutrition Agencies, Medicaid Agencies, and other State Agencies that play key roles in the DCM-F/RP process using a customized Excel-based cost log and clarification calls. We will ask respondents to report only costs that are in addition to costs for direct certification activities that were taking place before the DCM-F/RP demonstration. The sample for these cost logs includes 15 State Child Nutrition Agency staff, 15 State Medicaid Agency staff, and 3 other State Agency staff. Based on experience from the first DCM demonstration, we expect to achieve 100 percent response from State Child Nutrition Agency staff and other State Agency staff and 93 percent from State Medicaid Agency staff. In total, a 99 percent response rate is anticipated. To ensure this high response rate, we will track respondents who have completed cost logs, and send email reminders to those who have not. We will contact respondents who are slow in responding to cost log requests to answer their questions and remind them of the importance of the demonstration evaluation. If necessary, FNS staff may contact the most reluctant respondents to underscore the requirements of study participation.

Table B.1.3 presents sample sizes and expected response rates for each research activity.

Table B.1.3. Sample Sizes and Expected Response Rates, by Research Activity



Sample size


Expected number of participants

Respondents

Research activity

Agencies/
districts

Individuals

Expected response rate

Agencies/
districts

Individuals

State Child Nutrition Agency Staff

On-Site Interviews

15

45

100

15

45


On-Site Observations

15

30

100

15

30


Site Visit Follow-Up Telephone Interview

15

15

100

15

15


Administrative Records Request

15

15

100

15

15


State Cost Data Collection Tracking Logs

15

15

100

15

15

State Medicaid Agency Staff

On-Site Interviews

15

45

100

15

45


On-Site Observations

15

30

100

15

30


Site Visit Follow-Up Telephone Interview

15

15

93

14

14


State Cost Data Collection Tracking Logs

15

15

93

14

14

Other State Agency Staff

On-Site Interviews

3

3

100

3

3


State Cost Data Collection Tracking Logs

3

3

100

3

3

District Staff

On-Site Interviews

34

102

100

34

102


On-Site Observations

34

68

100

34

68


Site Visit Follow-Up Telephone Interview

34

34

91

31

31

Vendor Staff

On-Site or Telephone Interviewsa

4

4

100

4

4

Total Number of Responses


553

553

99

545

545

Total Respondent Sample


71

199

100

71

199

Note: State Child Nutrition Agency, State Medicaid Agency, other State Agency, and school district staff are all expected to participate in multiple data collection activities, including the site visits, follow-up interviews, and the completion of cost tracking logs and administrative records requests, where applicable. The “Total Number of Responses” row includes each relevant activity. Each individual is counted only once in the total respondent sample.

a Vendor staff interviews may be conducted by telephone, if vendors are not located near the State agencies.

B.2. Procedures for the Collection of Information

Describe the procedures for the collection of information including:

  • Statistical methodology for stratification and sample selection,

  • Estimation procedure,

  • Degree of accuracy needed for the purpose described in the justification,

  • Unusual problems requiring specialized sampling procedures, and

  • Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Data Collection

In this section, we provide information on the specific data collection activities for each key DCM-F/RP study component:

1. On-site interviews and observations. To document the DCM-F/RP processes, we will use in-depth case studies that trace the relevant direct-certification workflow step by step. The core of these studies will be in-person site visits in each participating State, during which we will interview program and technical staff at the State and district levels who are involved in implementing DCM-F/RP. We will first send introductory letters (Appendix A-1) and conduct scheduling calls (Appendix A-2) with State agency contacts to arrange the visits, then provide a site visit preparation document (Appendix A-2a through A2-c) containing the schedule and an overview of topics to be discussed and documents requested. Each site visit will last two or three days. During the visits, we will interview the staff at each State Agency, selected district, and vendors (where applicable) that play key roles in the DCM-F/RP process, and we will conduct observations of DCM-F/RP procedures. For example, we will ask Medicaid staff to show us how they identify eligible children within the Medicaid data, and ask Child Nutrition and district staff to show us how they conduct data matching. These observations (Appendix A-4) will ensure that site visitors have a complete understanding of how DCM-F/RP is conducted. Interviews (Appendices A-3a through A3c) will provide detailed descriptions of DCM-F/RP procedures at the State and district levels and specific changes needed to initiate DCM-F/RP. We will ask open-ended questions and follow up with adaptive probes based on the information provided by the respondent.

2. Site visit follow-up telephone interviews. We will also conduct telephone interviews (Appendices A-6a through A-3c) with respondents at the State Agencies and districts that we visited during site visits. The interviews, which will be preceded by a scheduling email (Appendix A-5), will take place near the end of the school year and provide updated information on how the DCM-F/RP demonstration has changed in each State since the site visits. Interviewers will ask about whether and how challenges identified during the site visit have been resolved and about any additional challenges that may have been identified since the site visits. In addition, the telephone interviews will provide information about whether the staff time and resources needed to conduct DCM-F/RP decreased as staff grew accustomed to the new procedures.

3. Administrative records request. We will collect district-level administrative data from State Child Nutrition Agencies for the DCM-F/RP implementation year SY 2017–2018 and, for States beginning the demonstration in SY 2017–2018, a baseline year before the demonstration.5 We will request the data through an email (Appendix B-1) detailing the information required (Appendix B-2). The administrative records will include data on both certification for school meal benefits and participation (meals served), for each district in the State, and the request will focus on data that States already collect, to the extent possible, as follows:

  • Certification data. We will collect data on certification status, method, and basis, including data elements reported on form FNS-742, plus the numbers of students directly certified for free meals and for reduced-price meals based on Medicaid. For the States that conduct their first DCM-F/RP match by the last operating day of October 2017, we will collect data for that point and for the last operating day of October in the baseline year. For the remaining demonstration States, we will request the data elements as of the point in time just after their first DCM-F/RP match (or a point after a reasonable portion of districts have conducted or triggered their matches, in States where districts determine the timing of matching) and for the same date during the baseline year. Where available, we will also collect additional administrative data on DCM-F/RP match results and prior certification information.

  • Participation data. We will collect from each State elements from the district-level data that are aggregated to complete form FNS-10, the Report of School Operations (approved under OMB # 0584-0594 Food Programs Reporting System, expiration date September 30, 2019).6 To assess participation, we will collect each district’s total numbers of reimbursable lunches and breakfasts served, by reimbursement category (free, reduced-price, paid) in each month during the demonstration and the same months during the baseline year. To facilitate analyses of Federal reimbursement costs, we will also request the numbers of meals reimbursed at the slightly higher “needs-based” NSLP rates or “severe-needs” SBP rates, and the number served in districts certified to receive an extra six cents per lunch served based on meeting updated nutrition standards and meal patterns.

We will collect these administrative data in two batches. The liaison for each State will contact the Child Nutrition Agency staff to first request the baseline data (in States that are new to the demonstration) and the certification data (for States that conducted their first DCM-F/RP match at the beginning of the school year). We will collect the data covering the first year of the DCM-F/RP demonstration when they are available. We expect most data to be provided in Excel files, but we will accept other formats. Because the data will not contain any personally identifiable information, the files can be submitted via email.

4. State cost data collection tracking logs. To analyze administrative costs related to implementing DCM-F/RP, we will collect data four times during the school year on resource expenditures from relevant State Child Nutrition Agency, Medicaid Agency, and other State Agency staff. These data will cover labor, infrastructure, software, and other costs. They will reflect additional costs States incurred to implement the new demonstration, beyond those associated with existing State efforts on direct certification.

To facilitate careful tracking, we will provide cost log templates (Appendices C-3a and C-3b) to States in Excel format at the beginning of each data collection period. Respondents will use the template to track and report monthly records of the time each staff member spends on various DCM-F/RP demonstration activities, and other direct costs incurred, such as amounts paid to outside contractors. We will also provide instructions for the template (Appendix C-2), describing the seven tabs, which include (1) a list of activities that might be involved in the DCM-F/RP process, tailored to Child Nutrition and Medicaid Agencies; (2) a log to enter monthly time for each staff type; (3) a table to enter salary information for each staff type; (4) a table to enter other direct costs; (5) a table to enter indirect costs; (6) a place to provide contact information; and (7) an optional tab for recording time weekly rather than monthly.

Liaisons from the evaluation team will collect the cost logs from each agency in their States and will include instructions with each data request, sent via email (Appendix C-1). After the completion of the first log, the liaison will conduct a clarification call (Appendix C-4) to ensure the provided data is accurately interpreted. The logs will be collected in four batches in the following months: (1) July through September, (2) October through December, (3) January through March, and (4) April through June. For States new to the demonstration in SY 2017–2018, the first log will also collect data on any costs incurred prior to July 2017. Logs will be collected the month following the end of the quarter, contingent upon OMB approval prior to that month, or else retroactively based on recall in the month following OMB approval.

Data collector training. Before data collection begins, liaisons from the evaluation team will attend a half-day training webinar. The webinar will provide background on the demonstration and detailed instruction in how to collect the different types of data described above. Data collectors are expected to have experience collecting similar data for direct certification studies, such as the previous DCM or DCM-F/RP studies.

Sampling and Analysis

1. Statistical methodology for stratification and sample selection

Quantitative analyses will be based on the full universe of states and districts participating in the DCM-F/RP demonstration. The only sampling will be to select districts for the qualitative data collection (site visits and follow-up telephone interviews). Districts will be selected purposively (1) to reflect variation along several relevant characteristics and (2) to facilitate logistics of site visits; the sample will not be statistically representative.

2. Estimation procedure

Qualitative analysis. To begin the analysis of qualitative data, we will draft an internal analytic memo after each site visit that will summarize findings from the in-depth interviews and observations. We will construct an analysis framework based on the relevant research questions and emergent themes from the analytic memos. We will use this framework to conduct a thorough analysis of all interview and observation notes and State documents, using either qualitative analysis software (such as NVivo) or detailed theme tables in Microsoft Excel. The analysis will identify patterns across different States and districts, focusing on matching procedures, challenges, and successes in implementing DCM-F/RP.

Quantitative analysis. We will use administrative records data to compute for each district measures of key certification, participation, and Federal reimbursement outcomes in the year before DCM-F/RP implementation and in the first year of DCM-F/RP. We will use quantitative pre/post district fixed-effects analysis to estimate the changes in certification that accompany DCM-F/RP implementation, controlling for measurable time-varying district characteristics (such as enrollment and local economic conditions) and all district characteristics that are fixed over time.

For other outcomes, we will use descriptive analyses. For example, we will analyze data on State administrative costs to estimate the total State-level administrative costs of the new demonstration, as well as breakdowns of these costs by agency, timing, and category (such as labor).

3. Degree of accuracy needed for the purpose described in the justification. There are no precision requirements for this evaluation. Quantitative analyses will be based on the full universe of states and districts participating in the demonstration, which will provide more accurate estimates than a smaller sample.

4. Unusual problems requiring specialized sampling procedures. There are no unusual problems requiring specialized sampling procedures.

5. Any use of periodic (less frequent than annual) data collection cycles to reduce burden. This data collection request is to study one year of the DCM-F/RP demonstration; all data collection will take place over the course of that year.

B.3. Methods to Maximize Response Rates and Deal with Non-Response

Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

Anticipated response rates and methods to maximize them are shown in Table B.3.1, below. Response rate assumptions are based on experiences with similar studies such as the first DCM demonstration. In that study, we obtained response rates of 100 percent on the collection of administrative records and State cost data tracking logs with a sample size of six State. The data collection for SY 2016–2017 of the DCM-F/RP demonstration is in progress, so response rates have not yet been computed. The first DCM demonstration did not include site visits or follow-up telephone interviews. We expect that the planned methods of data collection will result in the accurate, reliable data needed for the planned analyses and modeling at acceptable response rates. The number of completed instruments will be the numerator in response rate calculations. A completed instrument will be defined as one in which all critical items for inclusion in the main analysis are complete and within valid ranges. To maximize response rates and ensure the highest quality data possible, we will take a multipronged approach:

  • Site visits and follow-up interviews with respondents will be scheduled in advance in order to answer their questions and ensure that the data collection takes place at convenient times.

  • We will track respondents who have provided administrative records and completed cost logs and interviews, and send email reminders to those who have not.

  • We will contact respondents who are slow in responding to administrative records, cost log, and interview requests to answer their questions and remind them of the importance of the demonstration evaluation.

  • As appropriate, State Child Nutrition staff or FNS staff may contact the most reluctant respondents to underscore the requirement of study participation.

  • Staff conducting the on-site interviews and telephone interviews will be qualified, well-trained professional interviewers.

Table B.3.1. Expected Response Rates and Methods to Maximize Response Rates, by Research Activity

Respondents

Research Activity

Expected Response Rate

Methods to Maximize Response Rate

State Child Nutrition Agency Staff

On-Site Interviews

100

Scheduling calls


On-Site Observations

100

Scheduling calls


Site Visit Follow-Up Telephone Interview

100

Scheduling emails


Administrative Records Request

100

Email and telephone reminders


State Cost Data Collection Tracking Logs

100

Email and telephone reminders

State Medicaid Agency Staff

On-Site Interviews

100

Scheduling calls


On-Site Observations

100

Scheduling calls


Site Visit Follow-Up Telephone Interview

93

Scheduling emails


State Cost Data Collection Tracking Logs

93

Email and telephone reminders

Other State Agency Staff

On-Site Interviews

100

Scheduling calls


State Cost Data Collection Tracking Logs

100

Email and telephone reminders

District Staff

On-Site Interviews

100

Scheduling calls


On-Site Observations

100

Scheduling calls


Site Visit Follow-Up Telephone Interview

91

Scheduling emails

Vendor Staff

On-Site Interviews

100

Scheduling calls

Total Respondent Sample


100

193


B.4. Test of Procedures or Methods to Be Undertaken

Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

FNS received approval for pre-testing the data collection procedures and instruments for this evaluation under Approved Generic OMB Clearance No. 0584-0606 (approved on December 19, 2016). This request is to obtain clearance to conduct research with State Child Nutrition and Medicaid Agency officials and school district staff to develop, test, and improve evaluation data collection instruments and methodologies. We are pre-testing the site visits, cost logs, and administrative records requests, as well as supporting materials (including written instructions) in seven States in SY 2016–2017. A representative of the Child Nutrition Agency in each State is providing input related to administrative records data collection, representatives of both Child Nutrition and Medicaid Agencies in each State are providing input related to cost logs data collection, and representatives of the same agencies and at least one district in each State are providing input related to qualitative data collection.

Respondents have generally indicated that the process of scheduling site visits was not very difficult, the relevant staff participated, and the questions were clear. They also indicated that although it was sometimes challenging to classify certain types of costs, instructions were clear and comprehensive and completing the cost logs was not very difficult. Additionally, respondents have told us that the administrative data request was clear. Based on input from pre-test respondents and experiences during the pre-test data collection year to date, we have made the following revisions to data collection instruments:

  • Added materials to send to site visit contacts in each location before the site visit, including the agenda, a list of topics to be discussed, and a list of requested documents (Appendices A-2a through A-2c).

  • Added a few questions or probes to site visit protocols and the cost log clarification call protocol.

  • Revised text in the State administrative cost data collection materials to refer to “Medicaid eligibility agency” rather than “Medicaid agency” because in some States the agency that operates the Medicaid program differs from the agency that determines eligibility. The latter agency is typically most involved in direct certification.

  • Removed both the contact information tab and the option to provide initials rather than staff positions or job titles from cost logs, to address concerns from States about personally identifiable information.

  • Added two new activity categories to the cost logs.

  • Added a data element to the administrative data request form to ensure that States that cannot separate TANF, FDPIR, and other programs besides SNAP and Medicaid provide a total for those other programs.

In addition, we have reflected pretest experiences in the burden estimates shown in Appendix E.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Mathematica and FNS staff, as well as staff from Insight Policy Research, were consulted on statistical aspects of the design (see Table B.5.1). The same staff are proposed to be responsible for the collection and analysis of the study’s data.

Table B.5.1. Individuals Consulted

Mathematica Staff

Lara Hulsey

Project Director

609-936-2778

Joshua Leftin

Researcher

202-250-3531

Andrew Gothro

Researcher

202-250-3569

Daniela Golinelli

Senior Statistician

202-838-3597

Quinn Moore

Senior Researcher

609-945-6592

USDA Staff

Conor McGovern

FNS Project Officer (Research Analyst-COR/OPS/FNS)

703-457-7740

John Endahl

Senior Program Analyst/OPS/FNS

703-305-2127

Vivian Lees

Senior Technical Advisor/PMOSD/CN/FNS Child Nutrition Division

703-305-2322

Jamie Blair-Walker

Operational Support Branch, Child Nutrition Division

703-457-7751

Doug Kilburg

Mathematical Statistician, NASS

202-720-3777

Insight Policy Research Staff

Brian Estes

Senior Researcher

703-504-9492


1 Lara Hulsey, Joshua Leftin, Anne Gordon, Claire Smither Wulsin, Nicholas Redel, Allen Schirm, Nicholas Beyler, Brian Estes, and Carole Trippe. “Year 2 Impacts of Using Medicaid Data to Directly Certify Students for Free School Meals.” Final report submitted to the U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support. Princeton, NJ: Mathematica Policy Research, June 2016.

2 We originally assumed that the pretest would include 18 districts, but that was based on an assumption about the number of States in the pretest that conducted only local-level matching. When we later learned that an additional state conducted central-level matching, we reduced the number of districts included in the site visit to that state accordingly.

3 Seven of these States—California, Florida, Massachusetts, Nebraska, Utah, Virginia, and West Virginia—began conducting DCM-F/RP in SY 2016–2017, and eight States—Connecticut, Indiana, Iowa, Michigan, Nevada, Texas, Washington, and Wisconsin—will begin in SY 2017–2018.

4 With central matching, the State has primary responsibility for developing and maintaining the direct certification system.

5 The States that began the demonstration in SY 2016–2017 will have provided baseline data as part of the pre-test.

6 These data elements are not available at the district level in data that are already available.


File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified0000-00-00
File Created0000-00-00

© 2024 OMB.report | Privacy Policy