Findings and Recommendations from Cognitive Testing Interviews
Lisa Lee, Sr. Survey Methodologist
Laurie Imhof, Sr. Survey Director
February 29, 2012
Consumer Survey of Attitudes Toward the Privacy and Security Aspects of Electronic Health Records and Electronic Health Information Exchange
Office of the National Coordinator for Health Information Technology (ONC)
•Test draft English and Spanish versions of questionnaires
•Assess respondents’ comprehension of the questionnaire items, including question intent and the meaning of specific words and phrases
•Examine respondents’ retrieval of relevant information from memory; decision processes involved with answering a question; and appropriateness of response categories
•Provide information on average administration time of questionnaire
Purpose
•Participants were recruited from Chicago area by focus group facility
•Match socio-demographic characteristics of target population (oversample African Americans and Hispanics/Latinos)
•Total of 25 completed interviews (15 English; 10 Spanish)
•Interviews were conducted at NORC offices by survey methodologists and project staff
•Interviews were conducted January 12 – February 9, 2012
•Spanish interviews were conducted by native Spanish-speaking NORC survey methodologist
Methodology
•Administered verbal consent form
•Administered draft survey questionnaire over the telephone from separate conference room
•Administered cognitive interview protocol in-person
•Respondents received $40 incentive for their participation
Methodology (continued)
Respondent Demographics
•Gender
•Male: 11
•Female: 14
•Age
•18-25: 5
•26-35: 4
•36-45: 5
•46-55: 7
•56+: 4
•Race/Ethnicity
•White/Caucasian: 7
•Black/African American: 6
•Asian: 2
•Hispanic/Latino: 10
–All Spanish interviews
Note: Hispanic/Latino respondents were included in Spanish interview group only
•All interviews (25)
•Average administration time: 19 min
•English interviews (15)
•Average administration time: 19 min
•Spanish interviews (10)
•Average administration time: 20 min
–Note: Timings do not include introductory script or administration of verbal consent, which required an additional 3-6 minutes
Questionnaire Timings
•Current questionnaire is 5-10 minutes too long
•5-minute increase in administration time = 3% drop in the CASRO response rate and 13% increase in interviewer hours
•10-minute increase in administration time = 5% drop in the CASRO response rate and 33% increase in interviewer hours
-CASRO (Council of American Survey Research Organizations) response rate = Working residential number resolution rate * Household screening completion rate * Survey interview completion rate.
-Based on NORC findings from the 2011 National Immunization Survey, one of the largest national RDD landline and cell telephone surveys. Results are from questionnaire re-design experiments and the inclusion/exclusion of 5-minute modules.
Questionnaire Timings – Effect on Response Rates and Costs
•Goal is 10-15 minute questionnaire
•Average time for introductory script, selection of HH respondent and consent = 5 minutes
•Average time for questionnaire = 10-15 minutes
•Total survey administration time = 15-20 minutes
•Goal is to cut 5 to 10 minutes from questionnaire
•Average administration time is 2-3 questions per minute
•5 minutes = approx. 10-15 items
•10 minutes = approx. 20-30 items
•Goal = cut 20 items
Questionnaire Timings - Goals
Cognitive Interviews
Questionnaire Finding and Recommendations
First, let’s talk about your visits to a doctor or other health care provider.
I am interested in the visits you made to a doctor or other health care provider within the last 12 months. I want to know both the number of different doctors or other health care providers you visited and then the total number of visits.
First, how many different doctors and health care providers have you visited in the last 12 months? Please include primary care physicians, specialists, mental health professionals, physician assistants, nurses, clinics, and hospitals.
•1 to 2
•3 to 5
•6 to 9
•10 or more
•None
Question 1 –
Original Wording
•Question is lengthy due to the preamble.
•Suggest deleting preamble (paragraph beginning with “I am interested in the visits…”)
•Phrase “doctor or other health care provider” is awkward and lengthy to read over the telephone.
•Suggest shortening to “health care provider” throughout questionnaire and providing definition before first question.
Question 1 –
Findings
Reword question to:
First, let’s talk about your visits to health care providers. By health care providers, we mean primary care physicians, specialists, mental health professionals, physician assistants, nurses, clinics, and hospitals.
How many different health care providers have you visited in the last 12 months?
•1 to 2
•3 to 5
•6 to 9
•10 or more
•None
Question 1 –
Recommendations
Are you being treated for any chronic disease or medical condition such as, but not limited to, high blood pressure, diabetes, heart or lung disease, a mental health problem or arthritis?
•YES
•NO
Question 3 –
Original Wording
•Inclusion of the phrase “but not limited to” is not necessary. “Such as” conveys the point that the conditions listed are examples.
•Suggest deleting “but not limited to”.
Question 3 –
Findings
Reword question to:
Are you being treated for any chronic disease or medical condition such as high blood pressure, diabetes, heart or lung disease, a mental health problem or arthritis?
•YES
•NO
Question 3 –
Recommendations
How often do you receive a paper or electronic summary of your visit with a doctor or other health care provider?
•Never
•Rarely
•Sometimes
•Most times
•Every time
Question 6 –
Original Wording
• Some respondents indicated that they would include insurance and billing information as part of the visit summary, not just medical information.
•Suggest clarifying the question to specifically exclude non-medical information.
Question 6 –
Findings
Reword question to:
How often do you receive a paper or electronic medical summary of your visit with a doctor or other health care provider?
•Never
•Rarely
•Sometimes
•Most times
•Every time
Question 6 –
Recommendations
How comfortable would you feel asking for a copy of your medical information from your doctor or other health care provider?
•Very comfortable
•Somewhat comfortable
•Slightly comfortable
•Slightly uncomfortable
•Uncomfortable
•Very uncomfortable
Question 7 –
Original Wording
•Too many response categories. It is difficult to differentiate between “somewhat comfortable” and “slightly comfortable”. The scale should be revised to match other similar scales in the questionnaire.
•Suggest reducing the number of response categories and matching wording to similar questions.
•We need to clarify what “medical information” is being referenced in Q7. Is it the same as the “summary of your visit” mentioned in Q6?
Question 7 –
Findings
Reword question to:
How comfortable would you feel asking for a copy of your medical information from your doctor or other health care provider?
•Very comfortable
•Somewhat comfortable
•Not very comfortable
•Not at all comfortable
Question 7 –
Recommendations
How difficult was it to receive the medical information you requested? If you did not receive the information, please let us know.
•Not difficult at all
•A little difficult
•Somewhat difficult
•Very difficult
•I was not able to get the information I requested
Question 9 –
Original Wording
•The part of the question that reads, “If you did not receive the information…” does not need to be read. It can simply be included in the response options.
•Suggest deleting “If you did not receive the information, please let us know.”
•Change the last response option to “Or did you not get the information you requested?”, which will be read by the telephone interviewer as part of the response options.
Question 9 –
Findings
Reword question to:
How difficult was it to receive the medical information you requested?
•Not difficult at all
•A little difficult
•Somewhat difficult
•Very difficult
•Or did you not get the information you requested
Question 9 –
Recommendations
Are you currently caring for or making health care decisions for a family member?
•YES
•NO
Overall, how satisfied are you with the quality of the health care they have received from their doctor and other health care provider(s)?
•Very satisfied
•Somewhat Satisfied
•Dissatisfied
•Very dissatisfied
Section iii (Caregiving) – Q12 to Q17
Within the past 12 months have you asked for copies of their medical information from their doctor or other health care provider?
•YES
•NO
How difficult was it to receive the medical information you requested for your family member? If you didn’t receive the information, please let us know.
•Not difficult at all
•A little difficult
•Somewhat difficult
•Very difficult
•I was not able to get the information I requested
Section iii (Caregiving) – Q12 to Q17, continued
•What was the format of the copy of the medical information you received (paper or electronic)?
•RECEIVED AN ELECTRONIC COPY
•RECEIVED A PAPER COPY
•RECEIVED BOTH
•Have you accessed or used an electronic personal health record (PHR) on behalf of your family member? A PHR is an electronic application that allows you to access, enter, and manage medical information.
•YES
•NO
Section iii (Caregiving) – Q12 to Q17, continued
•The caregiving questions will require some respondents to consider more than one family member or more than one doctor when formulating responses to the questions.
•For the constructs being measured (e.g., level of satisfaction with quality of care and difficulty of receiving medical information) the correct response may vary for each family member or provider. For example, the respondent may be very satisfied with the care one family member received and very dissatisfied with the care another family member received. The respondent will have to form an answer that summarizes these different responses.
•These issues with the caregiving questions may increase burden on respondents and may make the responses to the questions difficult to interpret.
•Clarify the intent of asking the caregiving questions.
•Consider simplifying or deleting the caregiving questions.
•Relatively few respondents indicated that they were currently caring for/making health care decisions for a family member. It is possible that some respondents were not including minor children in their household. It may be necessary to explicitly mention children in the question.
Section iii (Caregiving) –
Findings/Recommendations
[ASKED ONLY IF RESPONDENT REPORTS NEVER ACCESSING THE INTERNET]
Could you access the internet easily if you needed to? (for example, at work, school, someone else’s house, a community center or library, etc.)
•YES
•NO
Question 19 –
Original Wording
•The current wording of the question does not capture the reasons why a respondent does not access the internet. For example, a respondent may have access to the internet but not want to use it. Somebody who does not use the internet may also not know how easily accessible it would be if they wanted to use it.
•Suggest clarifying the intent of the question.
Question 19 –
Findings
Revise to clarify question intent:
[ASKED ONLY IF RESPONDENT REPORTS NEVER ACCESSING THE INTERNET]
Is that because you do not want to use the internet or because it is difficult for you to access it?
•Do not want to use internet
•Difficult to access internet
Question 19 –
Recommendations
Have you ever kept an electronic personal health record, or PHR, for yourself? A PHR is an electronic application that allows you to access, enter, and manage some of your medical information.
•YES
•NO
Question 21 –
Original Wording
•Respondents were generally unaware of PHRs. Some respondents thought having a PHR meant being able to look at your medical information or test results online.
•Suggest including additional information (definition, example) of PHR.
Question 21 –
Findings
To the best of your knowledge, does your doctor or other health care provider:
•Electronically send prescriptions or refill requests directly to your pharmacy? YES NO
•IF NO: Should they be able to do this?
YES NO
•Electronically send your medical records to other providers that are caring for you? YES NO
• IF NO: Should they be able to do this?
YES NO
• Electronically obtain your medical records from other providers?
YES NO
• IF NO: Should they be able to do this?
YES NO
Question 25 –
Original Wording
•Some respondents answered these questions based on whether their providers had ever actually done each activity, not whether they had the capability.
•Suggest clarifying the intent of the question (capability to perform activity, not whether the provider has ever performed each activity for the respondent).
•Remove reference to “you” so that the questions pertain to the provider’s general capabilities, not what the provider has done for the respondent specifically
•The follow-up questions for respondents who answer “no” for each activity interrupt the flow of the question and make it more difficult to understand the next question in the series without repeating the stem. Question 24 should adequately cover the issues regarding whether medical information should be shared or sent electronically.
•Suggest deleting follow-up questions
Question 25 –
Findings
Revise wording to:
To the best of your knowledge, can your health care provider:
•Electronically send a patient’s prescriptions or refill requests directly to pharmacies? YES NO
•Electronically send a patient’s medical records to other providers that are caring for them?
YES NO
• Electronically obtain a patient’s medical records from other providers?
YES NO
Question 25 –
Recommendations
If your providers were able to share your medical information electronically, do you think it is very likely, somewhat likely, not very likely or not at all likely that each of the following is to occur?
a. The quality of the care that you receive would be improved.
b. The safety of the care that you receive would be improved
c. You would have fewer repeated tests because doctors and other health care providers and hospitals could access your recent test results.
d. Your doctors and other health care providers would do a better job coordinating your care with other doctors and health care providers.
• Very likely
• Somewhat likely
• Not very likely
• Not at all likely
Question 26 –
Original Wording
•These questions are very hypothetical and are fairly lengthy for a telephone interview. Items “c” and “d” in particular are complex. If the intent of the questions are to measure perceived benefits of health IT, other questions may already cover this.
•Suggest deleting questions
Question 26 –
Findings and Recommendations
Core Questions
How confident are you that you have some say in who is allowed to collect, use and share your medical information?
Having a say in who can collect, use and share your medical information has to do with the privacy of your records.
How confident are you that safeguards (including the use of technology) are in place to protect your medical records from being seen by people who aren’t permitted to see them?
Having safeguards (including the use of technology) in place has to do with the security of your medical records.
•Very confident
•Somewhat confident
•Not confident
Questions 27 and 28 –
Original Wording
Core Questions
•The question is presented first, followed by the definitions for privacy and security. Because the question has already been asked, respondents sometimes attempt to answer before the italicized text is read.
•The questions are a bit long.
•Italicizing part of the question text may make it unclear to interviewers whether the text should be read to all respondents or only read for clarification as needed.
•Suggest removing italicization from the question text.
•The parentheses in the question on safeguards may suggest to interviewers that the parenthetical text should only be presented as needed for clarification.
•Suggest removing parentheses from the question text and replacing with commas.
Questions 27 and 28 –
Findings
<number>
<number>
If you have looked online at your medical information, like lab tests, how confident do you feel that your medical information was kept safe and private?
• Very confident
• Somewhat confident
• Not very confident
• Not at all confident
• Not applicable/I have never looked online at my medical information
Question 29 –
Original Wording
•The question applies only to respondents who have looked at their medical information online. Suggest adding a skip instruction before Q29 based on respondent answers to Q20 to allow those respondents who have not viewed their medical information online to skip Q29
•If Q20 (Have you ever looked at any of your medical test results online) = YES, go to Q30.
•Rephrase “If you have looked online at your medical information, like lab tests” to apply to respondents who are known to have viewed medical information online.
•Remove the last response option: Not applicable/I have never looked online at my medical information
Question 29 –
Findings
Revise wording to:
How confident do you feel that your online medical information, like lab tests, was kept safe and private?
• Very confident
• Somewhat confident
• Not very confident
• Not at all confident
Question 29 –
Recommendations
Core Questions
If your medical information is sent by fax from one health care provider to another, how concerned are you that an unauthorized person would see it?
If your medical information is sent electronically from one health care provider to another, how concerned are you that an unauthorized person would see it? (Electronically means from computer to computer, instead of by telephone, mail, or fax machine.)
• Very concerned
• Somewhat concerned
• Not concerned
Questions 31 and 32 –
Original Wording
Core Questions
•Q31 question is confusing because faxes can be electronic. The distinction between this question and Q32, on sending medical information electronically, is unclear for those respondents who are aware that faxes can be sent electronically.
•Clarify question intent. What is the distinction being made between faxed and electronic transmittal of medical information?
•If the intent of the questions is to compare the security of paper records vs. electronic records, change Q31 to refer to “paper” records or sending records “by mail.”
•Q32 includes a definition for the term “electronically” in parentheses after the question is presented. The definition contrasts computers to telephone, mail and fax. However, Q31 and Q32 are contrasting fax and electronic/computer. Also, because the question has already been asked, respondents may attempt to answer before the definition of “electronically” is read.
•Suggest clarifying the definition of “electronically” in Q32 and including the definition within the question.
Questions 31 and 32 –
Findings
If your health information were accessed by someone who did not have permission, how likely do you think it is that any of the following would happen? Respond with very likely, somewhat likely, not very likely or not at all likely.
a. You would be discriminated against
b. You would experience personal or professional embarrassment
c. The information would be used to steal your identity
d. The information would be used to commit fraud
• Very likely
• Somewhat likely
• Not very likely
• Not at all likely
Question 33 –
Original Wording
•The question is hypothetical for respondents who have not had their health information accessed by someone without permission.
• Having health information accessed by “someone who does not have permission” is very general. Respondent answers to the question may vary greatly depending on how they interpret the phrase. The “someone” could be a personal acquaintance or a hacker, for example, and each of these persons would pose different risks to the respondent.
•It is unclear what items “c” and “d” mean. Identity theft and fraud seem very similar.
•Clarify intent of the question.
•Make the question more specific.
•Who is the “someone” that respondents should consider?
•What scenario would result in discrimination or personal/professional embarrassment?
•What is the distinction between stealing identity and fraud?
•Questions 27 through 32 also capture concerns about negative consequences from unauthorized access to medical records. Consider deleting Q33 if it cannot be clarified.
Question 33 –
Findings
I am going to read to you some safeguards that could be taken to protect your computerized medical record. If doctors and other health care providers followed these safeguards, on a scale of 1 to 5, how much safer do you think your medical information would be? With 5 being much safer, and 1 being not any safer.
a. Train employees on privacy and security
b. Notify you if your medical information falls into unauthorized hands
c. Allow you to review who has accessed your medical information
d. Allow you to make informed choices about how your medical information is collected and used
e. Prosecute and penalize those who violate privacy and security laws
Question 34 –
Original Wording
•The scale used in this question is different from others used in the instrument. The question required repetition because respondents had difficulty with the scale.
•Statements a through e are complex and difficult for some respondents to understand.
•Discuss complex elements of the question.
•Simplify the question by revising the scale and/or the statements.
•Consider deleting Q34 if it cannot be simplified.
Question 34 –
Findings
I’m going to read a list of some of the types of information that may be in your medical record. I’d like you to tell me if you prefer this information be kept separate from the rest of your medical record when it is being shared.
a. The medications you’ve been prescribed
Yes No
b. Results of genetic tests
Yes No
c. Results of HIV tests
Yes No
d. Results of test for sexually transmitted diseases
Yes No
e. Mental health diagnoses and treatment
Yes No
Question 35 –
Original Wording
•Since the question does not explain what the context is in which records are to be shared, it is hard to interpret respondent answers. The nature and scope of how the medical information is shared needs to be clarified.
•Clarify the intent of the question or consider deleting it.
Question 35 –
Findings
For each of the following types of information, please indicate whether you would want to limit sharing to: a) only the provider treating you for that condition, b) specific providers of your choice, or c) all providers treating you?
a. The medications you’ve been prescribed
b. Results of genetic tests
c. Results of HIV tests
d. Results of test for sexually transmitted diseases
e. Mental health diagnoses and treatment
• Only the provider treating you
• Specific providers of your choice
• All providers treating you
Question 36 –
Original Wording
•Respondents had difficulty making making the distinction between the different response options. They frequently forgot about the “for that condition” phrase.
•There is also the potential that respondents will give inconsistent answers across Q35 and Q36. Respondents could say in Q35, for example, that they would not want medications to be part of the record that is shared. They could then indicate in Q36 that all providers treating you should have information on medications.
•As with Q35, the nature and scope of how medical information is being shared is not clear.
•As it is currently worded, the question does not adequately get at the concept of differential consent.
•Clarify the intent of the question or consider deleting the item.
Question 36 –
Findings
Existing laws provide a reasonable level of protection for computerized medical records and medical information today.
Doctors and other health care providers have measures in place that provide a reasonable level of protection for computerized medical records and medical information today.
•Strongly agree
•Agree
•Disagree
•Strongly disagree
•Don’t know
Questions 37 and 38 –
Original Wording
•Respondents know very little about existing laws. Some mention the HIPAA law. Others say they have no particular law in mind when answering Q37.
•Respondents also know little about the security measures in place at their doctor’s office.
•Although respondents had little difficulty answering the questions, it is important to note that their responses are not based on knowledge of laws or their doctor’s practices.
Questions 37 and 38 –
Findings
Core Question
Now how much do you agree with these statements about support for the use of computerized medical records?
I want my doctors or other health care providers to use a computerized medical record to store and manage my health information despite any concerns I might have about privacy and security.
•Strongly agree
•Agree
•Disagree
•Strongly disagree
Question 39 –
Original Wording
Core Question
•The Spanish language version of this question is very long and difficult for respondents to understand.
• Consider simplifying the question.
Question 39 –
Findings
Core Question
Revise wording to:
Now how much do you agree with these statements about support for the use of computerized medical records?
Even though there may be privacy and security issues, doctors should store and manage patient information electronically.
•Strongly agree
•Agree
•Disagree
•Strongly disagree
Question 39 –
Recommendations
Core Question
Now how much do you agree with these statements about support for the use of computerized medical records?
I want my doctors or other health care providers to use a computer to share my health information with other providers treating me despite any concerns I might have about privacy and security.
•Strongly agree
•Agree
•Disagree
•Strongly disagree
Question 40 –
Original Wording
Core Question
•The Spanish language version of this question is very long and difficult for respondents to understand.
• Consider simplifying the question.
Question 40 –
Findings
Core Question
Revise wording to:
Now how much do you agree with these statements about support for the use of computerized medical records?
Even though there may be privacy and security issues, doctors should share patient health information with each other electronically.
•Strongly agree
•Agree
•Disagree
•Strongly disagree
Question 40 –
Recommendations
Do you live in an urban, rural or suburban area?
•URBAN
•RURAL
•SUBURBAN
Question 42 –
Original Wording
•Whether the respondent lives in an urban, rural or suburban area can be determined from the ZIP code, which is collected in Q41. One respondent had difficulty determining which response to pick. She lived in the city and did not know that the city would be considered an urban area.
•Also, some respondents who live outside the city limits, in “inner ring” suburbs, live in environments that seem quite urban.
•Recommend deleting question.
Question 42 –
Findings and Recommendations
| File Type | application/vnd.openxmlformats-officedocument.presentationml.presentation |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |