CHIPRASupporting Statement Part B -- Revised Feb 8-1

SUPPORTING STATEMENT

Part B

Evaluation of the Children’s Health Insurance Program Reauthorization Act of 2009 (CHIPRA) Quality Demonstration Grant Program

Version: February 8, 2012

Agency of Healthcare Research and Quality (AHRQ)

TABLE OF CONTENTS

B. Collections Of Information Employing Statistical Methods 3

1. Respondent Universe and Sampling Methods 3

2. Information Collection Procedures 4

3. Methods to Maximize Response Rates 5

4. Tests of Procedures 6

5. Statistical Consultants 7

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

The information collected under this request is not based on probability samples and may not be generalizable beyond the states included in the demonstration. There are no sampling, imputation, or other statistical estimation techniques. Interview subjects are selected purposively and fall into one of the following respondent types:

1. Key project staff (up to 4 respondents in each state). Key project staff will include the project director, project manager, principal investigator, and/or medical director. Grant applications and final operational plans will be used to identify individuals in these roles. As the individuals most involved in project design and oversight, key project staff will provide insight into the implementation of demonstration projects and relevant contextual factors, and identify lessons and implications as to the broad application and sustainability of projects.

2. Other implementation personnel (up to 16 respondents). Other implementation staff will be staff involved in the day-to-day implementation of grant funded projects and will include state agency employees, provider trainers, health information technology (HIT) vendors, and/or project consultants. They will be identified based on a review of grantee applications and final operational plans and in consultation with key project staff. Other implementation personnel are important to interview because they will provide insight into the opportunities and challenges related to key technical aspects of project implementation.

3. External stakeholders (up to 8 respondents in each state, interviewed in-person during site visits). Stakeholders will be familiar with CHIPRA projects and may serve on advisory panels or workgroups but will not be involved in day-to-day project implementation. They will be identified based on a review of grantee applications and final operational plans and in consultation with key staff. External stakeholders are included in the data collection effort because, although they will not have daily involvement in grant-funded activities, they have (1) strong interest in children’s care quality in Medicaid and CHIP, and (2) understanding of contextual factors that may help or hinder grant activities.

4. Health care providers (up to 12 respondents in each state, interviewed in-person during site visits). Health care providers will be actively participating in demonstration grant activities or serving as comparison practices. States will furnish a list of all providers. If more than 12 providers are included on the list, AHRQ’s contractor will use it to select respondents who meet a range of specified criteria, such as practice type, practice size, and so forth. Many of the strategies funded by the demonstration grants are expected to affect providers’ clinical and administrative activities and their satisfaction with their ability to provide high-quality care. Collecting interview data from providers who directly participate in grant-funded initiatives is therefore an important component of AHRQ’s evaluation. Collecting interview data from providers who do not participate in grant funded activities is similarly important, because such data will help AHRQ understand what would have happened in provider practices in the absence of demonstration grants. Providers in both groups will be asked about their experiences providing care to children in Medicaid and CHIP, coordinating with other providers, use of HIT, and provision of patient-centered care. AHRQ’s contractor will consult key project staff on the purposive selection of providers to be interviewed.

5. Medicaid and CHIP personnel in non-demonstration states (up to 5 respondents in no more than 9 states, to be interviewed by telephone). AHRQ, CMS, and the evaluation contractor will select a purposive sample of eight or nine non-demonstration states that share key characteristics with demonstration states. The types of characteristics we will consider may include population size, proportion of urban and rural areas, geographic region, whether applied for a CHIPRA quality grant, and whether has initiatives in quality measurement, provide-based delivery models, and health information technology.Respondents from these states will be knowledgeable Medicaid or CHIP personnel including the Medicaid/CHIP director, the Medicaid health-IT coordinator, and/or project directors for state medical home initiatives. The selected individuals will be similar to key staff interviewed in demonstration states in terms of seniority and role of responsibility within the state’s Medicaid or CHIP office. The rationale for interviewing these individuals is to enrich AHRQ’s understanding of how the CHIPRA quality grants contribute to improved care quality above and beyond other quality-related initiatives happening at the same time. (Examples of other initiatives include those funded by the HITECH Act, the Pediatric Quality Measures Program, and various medical home initiatives.) The non-demonstration state respondents and key staff will be asked similar questions about state context and non-demonstration activities related to quality measurement, HIT, and provider-based models of care delivery. Non-demonstration state respondents will not be asked about the CHIPRA quality demonstrations because they are not implementing them.

The total number of interviews that will have to be conducted to yield a comprehensive, multi-faceted understanding of project implementation will range considerably, from 20 to 40, depending on the number, scope and complexity, and nature of the projects in a given state. States listed in their grant applications to CMS the specific individuals they planned to involve in project design and implementation. AHRQ used these lists to determine the types of respondents to include in the proposed information collection and to estimate the maximum number of respondents per type (indicated in parentheses in the paragraphs above). The lists make clear, for example, that the number of key project staff ranges only from two to four per state, and the number of external stakeholders serving as project advisors ranges from about six to eight per state. More variable are the number of “other implementation personnel” involved in projects and the number of health care providers states are recruiting to participate in their grant-funded projects. Judging again from the state applications, we believe it will suffice to allocate evaluation resources for up to 16 interviews with other implementation personnel and up to 12 interviews with health care providers per state.

2. Information Collection Procedures

Semi-structured interviews conducted one-on-one with the respondent will be used for this data collection effort. The interview guides (included in attachments B, C, D, E and F) will be customized based on the scope and nature of projects in a given state. The protocols address detailed questions about project implementation that do not lend themselves to self-administered questionnaires or other quantitative data collection methods. The timing and format (in-person or telephone) of each interview is described below.

• Key project staff will be interviewed in person beginning March 2012 and again by telephone about one year later, in 2013. The interviews in 2012 will focus on fidelity to implementation plans and progress in executing planned activities. A follow-up interview, in 2013, will focus on implementation progress. Only this respondent group will be re-interviewed in the second round.

• Other implementation personnel will be interviewed in person beginning in March 2012.

• External stakeholders will be interviewed in person beginning in March 2012.

• Health care providers will be interviewed in person beginning in March 2012.

• Medicaid and CHIP personnel in non-demonstration states will be interviewed by telephone beginning in April 2012.

Interview appointments will be made well in advance of in-person visits and telephone interviews. All respondents will be sent the invitation request (included in attachment G) by email. AHRQ’s contractor will then follow up with the respondent every three days, alternating phone and email contacts. If the respondent does not respond to any attempts at follow up within three weeks and the contractor cannot identify a reason for non-response (e.g., the respondent is out of the office), the contractor will stop attempting to contact the respondent. Respondents who agree to be interviewed will be sent a confirmation email (included in attachment G) one week prior to their scheduled interview.

Unlike prospective respondents in demonstration states, those in non-demonstration states may have less incentive to participate in an interview. We will therefore ask the CMS project officer who oversees the grant program to send a letter of introduction to the Medicaid director in selected non-demonstration states. The letter will explain the purpose of the interviews AHRQ hopes to conduct, request the state’s participation, and ask the Medicaid director to identify appropriate interview respondents among state staff. AHRQ’s evaluation contractor will then contact specified individuals and schedule interviews using the process described above.

Quality Control Procedures. AHRQ’s evaluation contractor has designated a team of experienced qualitative researchers to collect and analyze interview data described in this statement. The team leaders will host a team training session so that all researchers involved in data collection employ uniform, high-quality methods and are thoroughly familiar with the data collection instruments. All interviews will be conducted by two-person teams (a lead interviewer and note taker) and they will be digitally recorded (audio only) if respondents consent.

The same team members will be responsible for data coding using the qualitative data software Altas.ti. They will follow a thematic coding scheme to be developed by the team leaders.

3. Methods to Maximize Response Rates

The in-depth interview data collection is not based on probability samples and is not meant to represent anyone other than the respondents. Therefore a response rate does not apply to this activity. However, in awarding grants to demonstration states, CMS stipulated that states cooperate fully in the cross-state demonstration evaluation, including participation in in-depth interviews. Given this, and AHRQ’s experience conducting other process evaluations, ARHQ expects a high level of participation from key project staff, other implementation personnel, and external stakeholders. To further ensure the cooperation of respondents, contractor staff will attempt to minimize individual burden and develop interview schedules that respect site constraints and pressures.

• Minimize individual burden. Willingness of respondents to participate in in-person interviews may hinge on the time these meetings require. To minimize the burden, guides are designed to gather information that is as complete as possible in as little time as possible. AHRQ’s contractor has developed separate discussion guides for each respondent type so that respondents are not asked about activities or issues that are not applicable to them or the state in which they work. In addition, interviewers will meet with interview respondents in person in their own offices or at a location of their choice. Telephone interviews with key project staff will be scheduled at a time that is convenient for the respondent.

• Develop interview schedules that respect site constraints and pressures. The project team will work with each site to determine logistics and a schedule for the in-person interviews. The schedule will avoid conflict with other activities and allow individuals to find time in their calendars to spend with contractor staff.

While AHRQ expects a high degree of participation from all respondent types, we expect providers may be less readily available for in-person interviews than other respondent types. AHRQ will offer additional accommodations to this respondent type to increase the likelihood of their participation. We will offer to meet with providers outside of clinical hours, restrict the interview to 30 minutes if 45 minutes is not acceptable, and conduct the interview by telephone if the respondent says that would be more convenient.

4. Tests of Procedures

AHRQ conducted pilot tests of the protocols for key project staff, participating providers, and Medicaid MCO representatives in August 2011. The key project staff and participating provider protocols were selected for pretesting because project staff and providers are the most essential respondent types to the study, and because those protocols are the basis for three others (external stakeholders, other implementation personnel, and comparison providers). The protocol for Medicaid MCO representatives was selected for pretesting because it is most unlike the other protocols and the agency wanted to verify its utility and relevance.

The pretests were conducted as individual telephone interviews with a total of seven respondents. (Because of limited resources and time, the agency could not conduct pretests in person, although the actual interviews will be in person.) Pretest respondents were selected to represent a range of demonstration states (Alaska, Utah, Florida, Illinois, Massachusetts, Oregon, and Pennsylvania) and activities in all five grant categories.

AHRQ’s objectives during the pretests were to assess whether (1) interviewers could collect the information needed in the allotted time, (2) respondents could readily understand and answer the interview questions, (3) interviews flowed sensibly from topic to topic, and (4) the questions seemed to yield thoughtful, candid responses. The pretests also were useful for identifying interviewer training needs.

The organization of the protocols attached to this supporting statement directly reflect the pretest results. (Because of the overlap in protocol content across respondent types, we applied the insights gained from pretesting the three protocols to those for the other respondent types.) Specifically, the protocol for key project staff consists of a set of general questions to be addressed to a principal investigator or medical director, and sets of category-specific questions to be addressed to other key project staff, such as a project manager or director. This approach ensures AHRQ will capture both broad, contextual information and specific, technical information while making the most effective use of each respondent’s time. All other protocols consist of core and supplemental sections. The core sections contain the high-priority questions that the pretests suggest most respondents will be able to answer in the allotted interview time. The supplemental sections contain lower priority questions that interviewers will be trained to select from if the respondent answers core questions in less than the allotted interview time.

The recruitment and confirmation emails attached to this supporting statement also reflect insights from the pretests. Specifically, pretest respondents suggested it would be helpful to know in advance: the types of questions they will be asked, information about confidentiality, identify of research sponsors, and the use of audio recording during interviews.

5. Statistical Consultants

AHRQ has contracted with Mathematica Policy Research, Urban Institute and Academy Health to conduct the evaluation of the CHIPRA quality demonstration grants. Table 1 identifies the individuals at these organizations who were consulted regarding the qualitative methods used in this project.

Table 1. Individuals Consulted Regarding Qualitative Methods of Evaluation

Name	Title	Email	Phone Number
Grace Ferry (Mathematica)	Research Analyst	gferry@mathematica-mpr.com	202-250-3571
Danna Basson (Mathematica)	Survey Researcher	dbasson@mathematica-mpr.com	510-830-3713
Cindy Brach (AHRQ)	Project Officer	cindy.brach@ahrq.hhs.gov	301-427-1444
Tennille Brown (AHRQ)	Project Officer	tennille.brown@ahrq.hhs.gov	301-427-1664
Rachel Burton (Urban Institute)	Research Associate	rburton@urban.org	202-261-5825
Kelly Devers (Urban Institute)	Senior Fellow	kdevers@urban.org	202-261-5905
Leslie Foster (Mathematica)	Senior Researcher	lfoster@mathematica-mpr.com	510-830-3709
Ian Hill (Urban Institute)	Senior Fellow	ihill@urban.org	202-261-5374
Henry Iryes (Mathematica)	Senior Fellow	hireys@mathematica-mpr.com	202- 554-7536
Lisa Simpson (Academy Health)	President and CEO	lisa.simpson@academyhealth.org	202-292-6747