National Health Service Corps
Chiropractic and Pharmacy Loan Repayment
Demonstration Project
Supporting Statement- PART B
The sampling design is a mechanism by which data from a subset of a population can be used to make inferences about the larger population. In probability surveys of finite populations, each unit in the population has a positive probability of selection that is known for selected units. These probabilities of selection are used to develop sampling weights, and the weighted sums of the observations are used to compute the survey estimates and variances of the population characteristics.
In an experimental setting, the design is defined by random assignment of the target population to treatment and comparison groups. Each randomly assigned member represents a replicate of the experiment, which enhances the external validity of this approach. These parameters and the sample sizes determine degrees of freedom and the power of estimates used to test treatment effects. Typically, the finite population inferences of the finite survey are not of interest in experimental design because the experimental units are assumed to be a random sample from a theoretical infinite population. Randomization of experimental units within the groups of a replicate is the basis for statistical testing.
In the evaluation of the demonstration to include chiropractic doctors and pharmacists in the NHSC loan repayment program, features of both survey and experimental design are involved. However, because the demonstration did not use random selection techniques to select the chiropractic doctors or pharmacists or to assign these professionals to clinics, the demonstration and the evaluation are based on what is referred to in the literature as a quasi-experimental design.
The first step in determining the respondent universe will involve the selection of clinics for the survey of clinic users and medical directors. The clinics where the chiropractic doctors and pharmacists are serving their NHSC obligations are community health centers in areas experiencing shortages of health professionals. There are 30 clinics participating in the demonstration; 7 have chiropractic doctors and 23 have pharmacists. Of these clinics, 20 will be selected for the surveys of clinic users and medical directors (5 with chiropractic doctors and 15 with pharmacists). Clinic selection criteria will include:
Clinic Size- Clinics must serve enough adult patients to support the survey of clinic users.
Comparison Clinics- At least two comparison clinics must be identified for each demonstration clinic.
Each demonstration clinic selected will be matched to two comparison clinics. The selection of comparison clinics will be based on statistical matching using propensity scores. Rosenbaum and Rubin (1983) showed that selection methods based on propensity scores produce a comparison group that is similar, on average, to the demonstration group along the characteristics used to generate the propensity scores. The propensity score is a single number that indicates the extent to which one clinic is similar to another along observed characteristics. Generating a propensity score requires three steps:
estimating a probability model using information on clinic characteristics for all demonstration and potential comparison clinics;
assigning a propensity score based on the clinic’s predicted probability from the estimated model; and
selecting potential comparison clinics with the closest absolute propensity scores, or the “nearest neighbor,” for each demonstration clinic.
MPR, the evaluation contractor, has successfully used this approach in other studies, including the evaluation of the State Partnership Initiative funded by the Social Security Administration. Currently, MPR is using this approach in the evaluation of the Home Health Independence Demonstration, funded by the Centers for Medicare & Medicaid Services.
Survey of Clinic Users- The sample for the survey of clinic users will be a census of adult users age 35 and older who come into the clinic during a two- to three-week study period and have used clinic services during the six months before the survey. The target number of completed surveys for each of the demonstration clinics is 40 and for each comparison clinic are 30, for a total of 2,000 completed surveys.
Survey of Clinic Medical Directors- The target population of the survey of clinic medical directors is the sample of medical directors at the demonstration and comparison clinics recruited for the survey of clinic users. The survey will sample the census of these directors, for a total of 60 completed surveys.
Survey of Chiropractic Doctors and Pharmacists Participating in the Demonstration- The target population for the survey of chiropractic doctors and pharmacists is the census of the chiropractic doctors and pharmacists participating in the NHSC demonstration. The survey will sample the census of the 7 chiropractic doctors and 23 pharmacists participating in the demonstration, for a total of 30 completed interviews.
Survey of Clinic Users- The survey of clinic users will use a census sampling strategy. Clinic staff will be instructed to select for the survey each adult patient age 35 or older that comes into the clinic for services during a specified period—for example, a specific two- or three-week period. Because community health centers are highly variable in the number of adult patients served and how appointments are scheduled, the implementation of the census sampling strategy will be tailored to the clinic. For example, time-interval sampling may be more appropriate for larger clinics that schedule appointments. In this approach, select time intervals (for example, a 30-minute interval or, perhaps, a 60- or 120-minute interval) are randomly selected and every adult patient who comes into the clinic during the time interval is selected for the survey.
After a patient selected for the survey checks in at the clinic’s reception desk, a clinic staff member will give the self-administered questionnaire to the patient and briefly explain the purpose of the study and that participation is voluntary. Respondents will have the opportunity to complete the survey in English or Spanish. Respondents will complete the survey in the waiting room, place the completed survey in the envelope provided, and return it to the reception desk before leaving the clinic. Completed surveys will be mailed to MPR to check for completeness, forwarded to a vendor for scanning and file preparation, and returned to MPR for analysis. The data collection process in 60 clinics is expected to require 3 months.
The data collection methodology is based primarily on a survey MPR conducted in food banks throughout the United States. The survey obtained information on demographic characteristics, income and other resources, and citizenship from clients served by agencies participating in America’s Second Harvest food banks. For the 2001 survey, 3,466 agencies (87.6 percent of those sampled) participated in the survey. Food bank staff and volunteers sampled 42,807 eligible clients, of whom 75.4 percent completed the questionnaires (Kim et al. 2001).1 MPR is currently conducting a second round of surveys.
Survey of Clinic Medical Directors- When each site is recruited for the survey of clinic users, the clinic’s medical director also will be asked to complete a survey. The field period will begin when a mail packet containing a cover letter, questionnaire, and postage-paid return envelope is mailed to the medical director. Because the cover letters and questionnaires vary, separate packets will be prepared for chiropractic demonstration sites, chiropractic comparison sites, pharmacist demonstration sites, and pharmacist comparison sites.
Two weeks after the initial mailing, a reminder postcard will be mailed. The postcard will be sent to all sampled clinics. The postcard will urge the medical directors to complete and return the survey, provide a toll-free number to call with questions or to request a new copy of the questionnaire, and thank those who may have already returned the questionnaire.
Two weeks after the reminder postcard is mailed (four weeks after the initial mailing), a full second mailing to nonresponders will be conducted. The mailing will mirror the initial mailing by including a cover letter, questionnaire, and return envelope. Two weeks after the mailing to nonresponders (six weeks after the initial mailing), executive interviewers, trained specifically for this study, will begin making follow-up telephone calls to collect survey data. Calls will be made to sample members who have not responded in an effort to collect the data by telephone or request that the survey be completed and sent in. Executive interviewers will also call responders for whom critical data items are missing, to collect this information by telephone or facsimile. Reluctant participants will be contacted by senior project staff to encourage participation. Telephone follow-up will continue for 4 weeks, completing the data collection process 10 weeks after the initial mailing.
A similar data collection approach was successfully used for the 340B Drug Pricing Program Survey. Copies of the cover letters appear in Appendix C.
Survey of Chiropractic Doctors and Pharmacists Participating in the Demonstration- A variety of information collection techniques will be used in conducting the Survey of Chiropractic Doctors and Pharmacists.
Identifying the Location of each Sample Member- MPR will begin the Survey of Chiropractic Doctors and Pharmacists by identifying the correct contact information for each person. The NHSC list of demonstration participants includes contact information for the clinic where the chiropractic doctor or pharmacist is serving the NHSC obligation. Each demonstration participant will be contacted to verify that the person’s contact information is correct. All relevant contact information will be entered into a database, which will allow the tracking of all contacts with demonstration participants.
Initial Letters to the Chiropractic Doctors and Pharmacists- A letter will be sent to each chiropractic doctor and pharmacist. A sample letter is included in Appendix D. The letter will briefly describe the project, the purpose of the interview, the time commitment involved, the topics to be covered, and how the interview relates to the other parts of the evaluation. In addition, the letter will indicate that someone will be contacting them within one week to schedule an interview. All letters will be reviewed and signed by NHSC.
Scheduling the Interviews- One week after the letters have been sent, a follow-up telephone call will be made to schedule the interview. An email message (or telephone call, if email is not available) will be sent to the chiropractic doctor or pharmacist the day before the scheduled interview to remind them of the interview.
Conducting the Interview- The interview will be conducted by a lead interviewer and a note taker. The lead interviewer will be responsible for asking the questions and ensuring a smooth flow and adequate discussion. The note taker will ask follow-up questions, as appropriate. The interviewers will introduce themselves and conduct the interview according to the Survey of Chiropractic Doctors and Pharmacists, which is included as Appendix D.
Developing the Interview Notes- The note taker will type the interview notes. The lead interviewer will review the notes and supplement them as needed. The notes will be formatted as text-only documents, to facilitate analysis using Atlas.ti, a qualitative analysis package. Atlas.ti is an off-the-shelf software package designed for code-based qualitative analysis that combines efficient management of non-numerical unstructured data with powerful processes of indexing, searching, and theorizing.
Survey of Clinic Users- The evaluation of the demonstration to include chiropractic doctors and pharmacists in the NHSC loan repayment program relies on a quasi-experimental design. The chiropractic doctors and pharmacists participating in the demonstration, as well as the clinics where they are serving their NHSC obligations, have self-selected into the demonstration. Therefore, the evaluation will rely on a parallel, or side-by-side, comparison of outcomes at demonstration and comparison clinics. Because the random assignment of clinics to demonstration or comparison groups is not possible at this stage, the alternative of selecting closely matched comparison clinics is used. As mentioned earlier, selection of comparison clinics will be based on statistical matching based on propensity scores.
In addition, the selection of a probability sample of clinic users is not feasible. Because of privacy and patient confidentiality issues, a sampling frame that lists all the recent clinic users age 35 or older is not feasible. Therefore, the objective of the evaluation is to assess program effects at demonstration and comparison clinics at a point in time. The data will allow us to present inferences for a hypothetical infinite population—that is, future potential users in underserved areas. The inferences will relate to potential effects, if any, which can be expected.
Survey of Clinic Medical Directors- Stratification is a feature of most sample surveys. In the survey of clinic medical directors, however, all the clinics that agree to participate in the survey of clinic users will be solicited for the survey of clinic medical directors. Hence, no sampling is proposed for this survey, and stratification is not relevant.
Survey of Chiropractic Doctors and Pharmacists Participating in the Demonstration- This survey will be a census, sampling and stratification is not proposed.
The information from the survey of clinic users will be used in both qualitative and quantitative analyses, while the information collected by the surveys of clinic medical directors and chiropractic doctors and pharmacists will be used in qualitative analyses only. The quantitative analyses will estimate the difference in clinic user outcomes between demonstration and comparison clinics. Ordinary least squares (for continuous dependent variables) and logistic regressions (for discrete dependent variables) within SAS will be used for covariance analyses. The models will use both the design variables for demonstration and comparison clinics, as well as other covariates to isolate the demonstration effects. For hypothesis testing and estimation, analysis will assume simple random sampling from an infinite population (sampling with replacement). The only adjustments that will be considered for these equal sampling weights will be to account for unequal sampling periods and nonresponse.
The goal will be 40 completed surveys at each demonstration clinic and 30 completed surveys at each comparison clinic, for a total of 2,000 completed surveys across 60 clinics. The analyses of the demonstration clinics with chiropractic doctors will be based on a sample of 500 clinic users, 200 from demonstration clinics and 300 from comparison clinics. With this sample, the evaluation will have sufficient statistical power to detect substantively important differences—that is, a minimum detectable difference between groups of 18 percentage points, based on an overall estimate of 50 percent, with 80 percent and alpha equal to 0.01. The analyses of the demonstration clinics with pharmacists will be based on a sample of 1,500 clinic users, 600 from demonstration clinics and 900 from comparison clinics. With this sample, the evaluation will have sufficient statistical power to detect substantively important differences—that is, a minimum detectable difference between groups of 10 percentage points, based on an overall estimate of 50 percent, with 80 percent and alpha equal to 0.01.
No unusual sampling problems are anticipated.
This is a one-time data collection.
Recruitment of Clinics- Achieving the highest response rates possible will depend on the clinics recruited for the surveys of clinic users and medical directors. Recruitment of clinics will begin when a letter from NHSC is sent to selected clinics to notify them of the study and encourage them to participate. The evaluation contractor will call the clinic administrators shortly after the letter is sent out to introduce the study and its purpose and goals.
After the initial call, MPR will send a written summary of the project and a description of the clinic’s role in the evaluation. In addition, clinics will receive a list of answers to frequently asked questions. Participation in the study will be encouraged in three ways:
The field procedures for the survey of clinic users are designed to be flexible and easy to incorporate into the clinic’s daily routines. During the recruitment of clinics, the evaluation contractor will work closely with each clinic to develop procedures that are easy for clinic staff to implement.
MPR will provide clinics with the support they need to implement the survey of clinic users. A toll-free number will be available if clinic staff has questions, and MPR staff will contact clinics regularly while the survey is being distributed to clinic users to ensure that any questions or concerns that clinic staff has are addressed immediately.
In return for participating, each clinic will receive a summary of the survey responses obtained from its clinic users. Many of the questions on the survey of clinic users request information about patient satisfaction and experiences obtaining care at the clinic. Clinics will gain some insight into how their adult users perceive the care they provide.
Survey of Clinic Users- The keys to achieving high response rates among clinic users are (1) designing questions that the target population can easily complete and (2) obtaining cooperation among clinic staff, which are responsible for implementing the sample design and distributing and collecting questionnaires from patients. The questions were based on the Consumer Assessment of Health Plans Survey (CAHPS). The CAHPS is a national survey designed for self-administration. Health plans, including those serving Medicaid enrollees, have used the CAHPS for several years. The CAHPS format has been developed and tested on a wide range of populations, including Medicaid enrollees, whose cognitive skills should be roughly comparable to patients visiting the clinics.
The sample and data collection procedures were developed to minimize intrusion on clinic operations. A key feature is sampling all eligible patients within time slots to avoid sampling from lists, which could be difficult to implement on days when clinics are busy. To facilitate compliance with instructions, a senior MPR interviewer or professional staff member will contact each clinic before fielding to review the sampling and data collection procedures. During the approximately two- to three-week data collection period at each clinic, an interviewer will contact the clinic periodically to verify that the questionnaires have been distributed and collected and to discuss problems. Overall, we are projecting a 75 percent response rate, which is the response rate obtained on America’s Second Harvest, a similar project described earlier.
Survey of Clinic Medical Directors- The data collection process for the survey of clinic medical directors follows well-tested procedures for mail surveys developed by Dillman (1999) and followed by many survey organizations.2 The cover letter that will accompany survey mailings will include the contact information for the project officer and a toll-free number at which to reach the MPR survey director.
Because the recruitment of clinics will include an agreement to complete the survey of clinic medical directors, we anticipate that all medical directors asked to complete a survey will do so. Respondents for demonstration clinics will be aware of the program and evaluation and its potential value and will be motivated to participate. Respondents from comparison clinics, however, may be less willing to participate, in spite of the initial agreement. If necessary, reluctant respondents will be contacted by senior project staff, including the survey and project director, and encouraged to complete the survey as agreed.
Survey of Chiropractic Doctors and Pharmacists Participating in the Demonstration- A four-step process will be used to obtain participation of all chiropractic doctors and pharmacists participating in the demonstration:
verifying the contact information for each person,
absorbing the scheduling burden by calling the respondents to arrange the interviews,
sending reminder email messages or placing reminder telephone calls the day before a scheduled interview, and
minimizing the burden on the chiropractic doctor and pharmacist by keeping the interview within an average of 30 minutes.
In addition to these steps, interviews will be scheduled at the convenience of the chiropractic doctor or pharmacist. They will be scheduled over lunch hours, in the evening, or during weekend hours to accommodate work schedules. Furthermore, the chiropractic doctors and pharmacists will be allowed to complete the survey during a series of calls if arranging a 30-minute block of time is not possible.
The data collection procedures described above will be replicated in pretests of each survey. At least one clinic with a chiropractic doctor serving an NHSC obligation through the demonstration and one clinic with a pharmacist serving an NHSC obligation through the demonstration will be selected for pretests of all three surveys. The pretest of the survey of clinic users will include fewer than nine adult clinic users.
The following individuals have contributed to the design of the surveys of clinic users, clinic medical directors, and chiropractic doctors and pharmacists participating in the demonstration:
Dr. Carol Irvin, an MPR senior researcher and project director for the evaluation (617-491-7900, ext. 232), provided overall direction for the project design.
Mr. Stephen Williams (919-467-4648), a senior statistician, provided guidance on the sampling designs and will prepare the survey samples and weights.
Mr. Richard Strouse (609-275-2332), an MPR vice president, designed the survey instruments and provided guidance on the data collection procedures.
Dr. Frank Potter, a senior fellow at MPR and sampling statistician, reviewed the clinic user sample design.
Dr. Randall Brown (609-275-2393), an MPR vice president, and Dr. Bob Schmitz (617-491-7900, ext. 236), an MPR senior fellow, reviewed clinic selection procedures.
Captain Evan R. Arrindell (301-594-4150), DSW, Chief of the Effectiveness and Preparedness Unit of the NHSC, within the Bureau of Health Professions of HRSA, is supervising the evaluation for the government.
In addition, two meetings of an expert panel were held to review and comment on data collection procedures, survey instruments, and proposed analyses. Table B.1 lists members of this panel.
Table B.1 Members of the Expert Panel for the Evaluation of the Demonstration to Include Chiropractic Doctors and Pharmacists in the NHSC Loan Repayment Program
|
|
Betty Chewning, Ph.D. Associate Professor School of Pharmacy University of Wisconsin-Madison 777 Highland Ave. Madison, WI 53705-2222 Phone: 608-263-4878 Email: bachewning@pharmacy.wisc.edu
|
George McClelland, D.C. Chairman of the Board American Chiropractic Association 155 Walters Dr. Christiansburg, VA 24073-1041 Phone: 540-382-3333 Email: macdc@usit.net |
Judith Cooksey, M.D., M.P.H. Associate Professor Department of Epidemiology and Preventive Medicine University of Maryland-Baltimore 660 West Redwood St. Baltimore, MD 21201-1596 Phone: 410-706-1277 Email: jcookseyumuic@aol.com
|
Freda Mitchem Director Systems Development and Policy Administration National Association of Community Health Centers, Inc. 7200 Wisconsin Ave., Suite 210 Bethesda, MD 20814 Phone: 301-347-0400 Email: fmitchem@nachc.com
|
Gary Gaumer, Ph.D. Assistant Professor Health Administration Simmons College 300 The Fenway, Room 338 Boston, MA 02115 Phone: 617-521-2373 Email: gary.gaumer@simmons.edu
|
Perry Pugno, M.D. Director, Division of Medical Education American Academy of Family Physicians 11400 Tomahawk Creek Parkway Leawood, KS 66211-2672 Phone: 800-274-2237, ext. 6700 Email: ppugno@aafp.org |
Christine Goertz, D.C., Ph.D. Director of Clinic Research Samueli Institute 1700 Diagonal Road Suite 400 Alexandria, VA 24073-1041 Phone: 703-299-4800 Email: cgoertz@siib.org |
Jon Schommer, Ph.D. Associate Professor Department of Pharmaceutical Care and Health Systems University of Minnesota College of Pharmacy 308 Harvard St., SE Minneapolis, MN 55455-0343 Phone: 612-626-9915 Email: schom010@tc.umn.edu
|
Eric Hurwitz, D.C. Assistant Professor in Residence Department of Epidemiology UCLA School of Public Health Box 951772 41-275 Center for Health Sciences Los Angeles, CA 90095-1772 Phone: 310-206-4074 Email: ehurwitz@ucla.edu |
Steve Wilhide Former Executive Director National Rural Health Association 5193 Brawner Place Alexandria, VA 22304 Phone: 703-519-7910 Email: wilhide@NRHArural.org |
1Kim, Myoung, Jim Ohls, and Rhoda Cohen. “Hunger in America 2001: National Report Prepared for America’s Second Harvest.” Princeton, NJ: Mathematica Policy Research, Inc., October 2001.
2 Dillman, Don. Mail and Internet Surveys: The Tailored Design Method. New York: John Wiley Company, 1999.
| File Type | application/msword |
| File Title | COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS |
| Author | HRSA |
| Last Modified By | HRSA |
| File Modified | 2006-08-02 |
| File Created | 2006-08-02 |