ANNEX D
TECHNICAL REVIEW PANEL MEETING REPORT
Survey on Rural Community Wealth and Health Care Provision
Technical Review Panel Meeting Report
February 2013
Background
On February 7, 2013, a Technical Review Panel (TRP) for the proposed survey met at the Economic Research Service (ERS) with the study team and a representative of the Office of Management and Budget (OMB) to review and discuss the proposed survey purpose, research questions and methods, and to provide advice to the study team on these issues. The list of TRP members and other participants in the meeting is provided in Annex A.
The issues discussed during the meeting included the purposes of the study, the research questions, the target population, the sampling approach, the draft questionnaires, and the statistical power of the analysis. The main issues discussed and the recommendations provided by the TRP and OMB representative are summarized below.
Study purposes
The primary purpose of the study, as stated by ERS, is to provide information about how rural small towns can attract and retain health care providers, considering the broad range of assets and amenities that may attract providers. The secondary purpose is to provide information on how improving health care may affect economic development prospects of rural small towns. ERS seeks to address these purposes by obtaining the perspectives of health care providers and community leaders in rural small towns. ERS considers these purposes to be primarily descriptive; the survey is not intended for the purpose of impact evaluation. ERS intends to complement the survey by econometric analysis of longitudinal county-level secondary data.
The TRP members agreed that the proposed survey addresses an important knowledge gap. No alternative purposes for the survey were suggested during the meeting. Related to the primary purpose, it was noted that various grant programs and agencies, such as Area Health Education Centers (AHECs), assist rural communities in recruitment and retention of health care providers. One possible focus of the survey could be to compare communities that have received assistance in recruitment and retention with those that haven’t. It was suggested that to achieve the primary purpose, it is important to know what rural communities are doing with regard to recruitment and retention. One comment was that these purposes and the associated research questions are not only descriptive, and that there is a need to consider what sources of information in addition to the survey are best suited to address the purposes and research questions. One panel member questioned why the focus was on health care in general; ERS clarified that the focus is on primary health care.
Research questions
The research questions proposed for the study are:
What community level factors (e.g., community assets, amenities) attract health care providers to practice in rural small towns? How important are these different factors to health care providers?
What factors affect whether health care providers decide to keep practicing in rural small towns, and how important are they?
To what extent is recruiting and retaining health care providers seen as an important priority by local health care providers and community leaders?
What efforts and investments do rural communities make to recruit and retain health care providers? How effective are these efforts, from the point of view of community leaders?
What major changes in health care availability and quality have occurred in the community in the past 10 years, and have changes in the recruitment and retention of health care providers affected this, from the perspective of health care providers and community leaders?
How are changes in health care availability and quality perceived to have affected other aspects of community economic development, such as the ability to attract or retain young families, retirees, or businesses?
Several comments and questions about the research questions were discussed:
It is also important to know the barriers that prevented providers from coming to rural communities. (This is considered among the factors affecting whether providers locate and stay in rural small towns).
The assumption that this is a priority for all rural communities is problematic. (Questions 3 and 4 do not assume that recruiting and retaining providers is a priority; whether that is so is part of these questions).
It could be useful to ask about anticipated changes in the next 5 years.
How do these questions map to the target populations of the survey? Are the target populations the appropriate ones to answer these questions?
What is the capacity of proposed survey respondents to respond to these questions; e.g., to questions about the past 10 years?
What other sources of data are available to answer these questions?
What is the level of analysis? Community or individual provider for provider survey? Community or individual leader/stakeholder for stakeholder survey?
Target population
The proposed target population includes primary health care providers and community leaders/other stakeholders in 809 rural small towns (population 2,500 to 20,000) in nine states in the Mississippi Delta (MS, LA, AR), Southern Great Plains (TX, OK, KS), and Upper Midwest (IA, MN, WI) regions. The small towns included in the target population were identified by 2000 Zip Code Tabulation Areas and rural urban commuting area codes. “Primary health care providers” are defined as the following types of practitioners with advanced (beyond a B.Sc.) degrees: primary care physicians (MD or DO), physician assistants, nurse practitioners, certified nurse midwives, dentists and pharmacists. “Community leaders and other stakeholders” are defined as those involved in the local health care industry, community economic development, or recruiting and retaining health care providers. Types of community leaders/other stakeholders include health care facility administrators, local government officials, business leaders, and representatives of nonprofit organizations involved in health care or community development.
Several issues were discussed regarding the target population of health care providers:
Need to define primary care providers carefully. For example, should consider including obstetricians/gynecologists as primary care providers. Internal Medicine can function as either primary care or a specialty.
Why include dentists and pharmacists in the target population? Dentists are quite different than physicians; more entrepreneurial. Pharmacists are less independent.
What are the boundaries for the providers included in the population? Will providers who live outside the town but work in the town be included? (Yes). Will providers who work outside the town limits but serve people in the town be excluded? (That is our intention, but we may reconsider this). What about providers who don’t have an office in the community, but serve the town on a part time basis? (We plan to include them). Might consider expanding the definition of the study community beyond the town limits, perhaps based on information collected from key informants.
How will coverage of the target population be ensured and assessed? We discussed ways of doing this, including using telephone calls with knowledgeable informants to check the provider list, cross checking lists from different sources, and checking this in selected towns during the second phase site visits. It was suggested that measuring coverage by type of provider would be useful. Another suggestion was to cross validate health care provider lists with key informants during pilot studies and derive some indication/measurement of health care provider coverage.
The National Association of Physician Assistants (PAs) is developing a list of PAs. Keith Mueller at the University of Iowa is developing a list of pharmacists. These lists might be useful in developing the frame.
Need to be clear that we are missing the population of providers who considered working in rural small towns but decided not to, and the population who came but left. Consider how this affects our ability to answer the research questions. Despite this limitation, it can be useful to compare responses of providers that are resident in the community vs. those who only work there, full vs. part time, with or without an office there.
May be able to talk to some providers who left selected communities in the second phase field visits. (The second phase field visits were described briefly but not included in the original ICR, and will be incorporated into the new ICR).
Several concerns and suggestions were raised about defining and identifying the target population of community leaders/other stakeholders:
This population is difficult to define. Need to clearly define who is in the population; e.g., hospital and clinic administrators, the town mayor and city council, and Chamber of Commerce.
Using an adaptive sampling approach will be difficult to implement, and it might not be clear what the resulting population represents.
Input from the community perspective is important, but it is only useful if respondents have relevant knowledge. The types of people who are knowledgeable may vary from one community to another, which adds to the difficulty of clearly defining the target population.
What if no one in the community is involved in recruiting or retaining providers? Who will be included in the stakeholder population in that case? Therefore the stakeholder population should not be defined by involvement in recruitment or retention efforts.
It was suggested that the focus should be on a small number of key informants in the community. This will likely include government officials involved in community development and heads of medical facilities such as hospitals and clinics.
Sampling
The target population will be sampled using a two-stage stratified sampling approach. In the first stage, 150 rural small towns have already been selected, stratified by the three study regions and whether the town has a hospital. Within each stratum, all towns were sorted by the number of primary care physicians in the town and population size, and the towns were selected using systematic sampling with a random start. The number of towns selected within each stratum was determined by minimizing the variance of the mean estimator of the number of physicians per 100K population in each town.
In the second stage, respondents are proposed to be selected using stratified simple random sampling of the populations of providers and community leaders/other stakeholders. The strata will be based on the types of providers and types of community leaders/other stakeholders. A maximum of 8 providers and 8 community leaders/other stakeholders will be sampled in each sample town.
A few issues were raised concerning the sampling approach:
Upon what years of data is the sample frame based? (Various years: the ZCTAs were classified based on 2000 census data, whether the town has a hospital is recent information, population size of the towns is for 2008, the number of primary care physicians and primary care physicians per population are for 2007). It would be useful to evaluate the status of the sample communities using data from the 2010 census.
Why not stratify by town population size or population density? (We sorted by town population size prior to systematic sampling to ensure representativeness along this dimension). Think about how the sampling approach will affect our results.
Questionnaires
Several issues were raised in discussing the questionnaires.
Questions and comments related to both questionnaires:
How do the survey questions map to the research questions?
The emphasis of the questionnaires is on collecting perceptions data. How useful will such data be for answering the research questions and for policy purposes? (Study team noted that perceptions matter regardless of objective reality and that for some measures, perceptions can be compared to secondary data).
Need some objective information on what communities actually have with regard to access to health care services and what they are doing with regard to recruiting and retaining health care providers. The utility of the dataset can be enhanced by showing how secondary data will be combined with the survey data. (The team will develop a format for collecting secondary data to be combined with the survey data).
It may be useful to compare perceptions of providers vs. stakeholders.
On community assets (Question 9 in provider questionnaire (PQ9), Question 15 in stakeholder questionnaire (SQ15)): Can some of this information be better obtained from other sources? Rather than perceptions data, can more objective indicators be collected? (We intend to collect data on assets from secondary sources as well, but such data are limited at the town level. Most of these items were included in this question series because obtaining objective measures would be difficult.) Some relevant secondary data may be available from the census of local governments.
Most of the questions in PQ9/SQ15 are phrased with a positive tone. It would be better to use neutral statements, or have a better mix of both positive and negative statements.
Compound questions should be revised or broken into multiple questions.
Consider using preliminary focus groups to identify response options for open ended questions, so some of them could be replaced with close ended questions.
PQ9c/SQ15c: It should say, “This town invests enough in …”
Why have a 10 year reference period for the questions about changes in health care? For what research purposes and questions is this necessary? It may be difficult for respondents to recall changes over such a long period, and some respondents will not have lived or worked in the community for 10 years.
Clarify that questions on changes in health care refer to primary care.
PQ14e and PQ16d/SQ21e and SQ23d: Mention local government policies vs. other levels. It would be good to have a follow up question about what policies.
Provider questionnaire:
How much time will be required for the provider survey? Concern about the survey length.
Is a telephone survey the best mode to use with providers? NCHS has had some good response from physicians using a web-based survey. However, NCHS conference proceedings recently released stated that paper mail surveys are most effective with physicians, with less success from web-based or telephone efforts. Offering concurrent survey modes typically reduces response rates, so perhaps a consecutive dual-mode approach is best.
Stakeholder questionnaire:
Recruitment questions – What about the role of large health care organizations in recruiting providers, either away from or to rural communities?
Statistical Power
Statistical power was estimated assuming binary response variables. Almost all of the closed ended questions in the survey are either binary response or ordinal response variables, and ordinal responses can be converted to a set of binary responses. The power calculations were illustrated with an example from PQ30 (which is critical to answer research question 1), with the range of power shown for alternative levels of key parameters (true population mean, effect size, intracluster correlation, and sampling fraction per community). These estimates showed that the statistical power was above 90% in almost all cases considered.
Several issues were discussed concerning statistical power:
The issue is not just whether there is adequate statistical power, but also whether the sample size proposed is necessary or optimal given the precision requirements for the estimates that the study intends to produce, and whether it would be better to have more communities or more respondents per community. For example, would it be better to sample only 100 towns, and possibly more respondents per town?
For community level analysis, the power will be lower since it is the same as the case when the intracluster correlation = 1.
Did the power analysis allow for nonresponse; i.e., fewer than 8 respondents in a community? (We can calculate the power as a function of number of respondents, instead of sampling fraction in each community).
It would be useful to consider comparisons across domains in the power analysis. (This was done in the original ICR Part B, using comparisons between regions and between towns with/without a hospital).
For variance estimation, may be able to get some figures from the surveys supported by the Robert Wood Johnson Foundation.
Recommendations from the TRP and OMB
Drop the community leader/stakeholder survey, and replace it with key informant interviews with a few knowledgeable people in each town, such as the hospital or clinic administrator, a senior local government official (mayor or town manager), and someone from the Chamber of Commerce. The selection of key informants can be purposive since no claim will be made that subjective perceptions collected are representative of a larger population.
Use a semi-structured interview instrument with the key informants, rather than a structured questionnaire. The key informant interviews can be used to collect or verify objective information on some topics, such as what health care facilities and providers are available in or near the community, how these have changed in the recent past, and what recent efforts have been made to recruit and retain health care providers. Both successful and unsuccessful efforts should be discussed. These interviews can also collect perceptions of the factors affecting recruitment and retention of providers and how changes in health care access have affected community economic development. This information can help in developing the sample frame for the provider survey and the provider questionnaire, as well as providing objective information needed to answer the research questions.
The findings of the key informant interviews can also be used to help plan the second phase of the study, which will involve field visits to selected communities.
Include pilot phases of the study to investigate the usability, feasibility and cost of the proposed information collection; determine whether the data collected will be adequate to answer the research questions; inform the provider sample frame and sampling approach; and inform the content of the provider survey. After completing each phase, provide a report to OMB for review, and then adjust the proposed approach as necessary. For example, we might propose:
Conduct key informant interviews in 10 pilot communities. Review results, and provide Report 1 to OMB, describing any proposed changes in the key informant interview approach, provider sample frame/approach, and provider questionnaire. May need to resubmit revised provider questionnaire for approval. If OMB recommends, an additional pilot of key informants in 40 communities will be implemented before proceeding with provider surveys. See Note below.
If OMB approves Report 1, conduct provider surveys in 10 pilot communities. Review results and provide Report 2 to OMB, describing any proposed changes in the provider sample frame/approach or questionnaire. May need to submit revised provider questionnaire for approval. If OMB recommends, an additional pilot of providers in 40 communities will be implemented before proceeding. See Note below.
While the provider surveys are being conducted in the 10 pilot communities, proceed with conducting key informant interviews in the remaining communities. Review results to assist in planning the field visit stage of the study.
If OMB approves Report 2, conduct provider surveys in remaining 140 communities. Review results to assist in planning the field visit stage of the study. Provide Report 3 to OMB, proposing selected communities for the field visit stage of the study, who will be interviewed, and revised formats for the visits.
If OMB approves Report 3, conduct the field visits in five pilot communities. Review results and provide Report 4 to OMB.
If OMB approves Report 4, conduct the field visits in the remaining selected communities.
Note: Depending on the recommended changes identified by the 10-community pilot, a second pilot phase with 40 communities could be incorporated into the process, with an additional report to OMB after that portion is completed. If OMB approves the additional report, the final 100 communities would then be completed. This second pilot could occur either after the key informant pilot of 10 communities (based on Report 1) or after the physician pilot of 10 (based on Report 2). Note also that, if the recommended changes to study scope or protocol are substantial enough, it may require a new Federal Register notice and revision or resubmission of the Information Collection Request (ICR).
Next steps/process going forward
Develop and publish new 60 day Federal Register notice, with input from OMB.
Develop and submit new ICR, after input from OMB. Include all phases of the research in the ICR, including the field visits. Include this report on the TRP meeting with the submission.
If OMB approves ICR, initiate above steps, starting with the key informant interviews in pilot communities.
Annex A. Participants in TRP meeting
Name |
Affiliation |
Role in Meeting |
Curt Mueller |
Office of Rural Health Policy, HRSA |
TRP member |
Cynthia Robins |
Westat |
TRP member |
David Woodwell |
National Center for Health Statistics |
TRP member |
Jennifer Park |
Office of Management and Budget |
OMB representative |
Pheny Weidman |
Economic Research Service |
ERS PRA clearance officer |
John Pender |
Economic Research Service |
Study team leader |
Shirley Huck |
Survey and Behavioral Research Services (SBRS), Iowa State University |
Study team member |
Jan Larson |
SBRS, Iowa State University |
Study team member |
Sarah Nusser |
Center for Statistics and Survey Methodology, Iowa State University |
Study team member |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | %USERNAME% |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |