Annex E - Report on Cognitive Interviews

ANNEX E

REPORT ON COGNITIVE INTERVIEWS

ERS: Rural Community Wealth and Health Care Provision

Cognitive Interviews for Provider Surveys

and Key Informant Semi-Structured Interviews

9/27/2013

The development of new survey instruments is a complex task. It can be challenging to ensure that survey questions are clear, concise, and universally understood by a variety of respondents. In addition, complex surveys or project protocols can be confusing to respondents, even if individual questions are clear. To assist with the development of the survey instruments for this project, two series of cognitive interviews were conducted, the first in February-March of 2012 and the second in September of 2013.

Series 1 Cognitive Interviews

The first series of cognitive interviews was conducted in preparation for an earlier Information Collection Request (ICR). Seven participants were recruited by word-of-mouth from small communities in Iowa that were not part of the project sample. Each cognitive interview participant fulfilled a role in his/her community similar to eligible respondents in the actual project. The participants included providers (physician, nurse practitioner, dentist) and key stakeholders in the community (medical facility administrators, economic development and community leaders). The cognitive interview participants included both men and women.

Methodology. The cognitive interviews were conducted over the telephone by SBRS staff using paper documents. A written script was used to introduce each interview and to provide consent information emphasizing confidentiality and voluntary participation. The cognitive interview participants were given basic background information about the project that was similar to the information that actual project participants will receive.

Respondents were asked to think aloud as they considered and processed the survey questions and to ask the interviewer if there was anything unclear or that they did not know how to answer. Throughout the interview, each survey question was read to the respondent and answers were recorded following standard interviewing procedures. Interviewers made notes to record respondent comments or questions. After all questions were answered, the interviewer probed for problems, pertinent issues that were not included, and general comments or suggestions.

Participants were sent a $50 gift card for a local grocery store to thank them for their time and cooperation. For documentation purposes they were also sent a receipt to sign and return to SBRS in an enclosed pre-addressed postage-paid envelope.

Results. The cognitive interviews identified no major problems with the surveys. However, numerous minor issues were raised. Potential revisions were discussed and carefully considered by the research group before changes were incorporated. The goal was to make the surveys clear and consistently understood for as many people as possible, and clarifying a text for one person could make it more obscure for others.

Several question transitions and introductions were revised for clarity. Minor edits were made to clarify several questions. Two items identified as redundant were deleted, and three new items were added. In the provider survey, questions relating to type of practice and residency or internship location were changed to clarify and focus more on the question intent. In addition to the survey revisions, the cognitive interviewing process also highlighted the need for certain resource documentation to be available for telephone research interviewers during project data collection.

Series 2 Cognitive Interviews

In preparation for the current ICR submission, four more cognitive interviews were conducted in September of 2013. The current submission involves mail/web surveys completed by health care providers and semi-structured interviews conducted with key informants. Most of the questions are subsets of the original surveys and had already been reviewed through the 2012 cognitive interview series. As a result, the primary goal of the 2013 cognitive interviews was to test format and procedure, with a secondary goal of verifying that the questions are clear and understandable.

The procedure for the mail/web survey cognitive interview was to ask three providers to complete the paper survey as they normally would, keeping track of the time required to complete it. Then SBRS staff asked the providers for feedback on the questions and format, a page at a time. There were no major problems identified, although the routing instruction for one question was changed. The participants indicated that they probably would prefer completing the survey online, however they liked seeing the paper copy so they knew exactly how long it was and what it included.

The key informant procedure was tested in one community by talking with a former mayor who has continued to be involved in local economic development since leaving office. The process went smoothly and indicates that the protocol seems sound. The semi-structured process seems to effectively accommodate overlapping responses and issues unique to individual communities. He indicated that the processes of verifying local providers and health care facilities should work well.

Conclusion. The cognitive interviews conducted by SBRS staff proved to be a valuable tool in the refinement of the survey instruments to be used for this study. Several clarifications and revisions were made that will increase the effectiveness and accuracy of the instruments. The research staff felt that the time and effort put into the series of cognitive interviews was very worthwhile and will result in greater consistency and accuracy of data.

3