Download:
pdf |
pdfField Testing Evaluation of the
Pregnancy Risk Assessment Monitoring System (PRAMS) –
Phase 9 Survey
Prepared for
Council of State and Territorial Epidemiologists (CSTE)
Centers for Disease Control and Prevention (CDC), Division of Reproductive Health
Prepared by
CareHalo Research Center (CRC)
EXECUTIVE SUMMARY
The purpose of this analytical report is to highlight respondents’ experiences with the
maternal and child health questionnaire (core) during field-testing. Analyzing the
summary of interviewer’s report of the ten respondents that participated in the fieldtesting was the focus of this study. The method was mixed and multi-phased, including
the preliminary coding of qualitative data, measurement of variables, univariate and
bivariate analyses of quantitative data, the processing of selected codes and the graphical
representation of data.
The qualitative data yielded six convergent and four divergent codes. The analysis of the
resultant numerical data from the measurement of the contents of some of these codes
shows that most respondents (7, 70.0%) reported that instructions were clear. A large
proportion of respondents (9, 90.0%) reported no difficulty in comprehending the
questions while all respondents (10, 100.0%) stated that choosing answers was easy for
them. Furthermore, bivariate analyses of the data indicates that there is no significant
association between respondents’ spoken language, education, race and perceived
quality of instructions, comprehension of questions as well as making choices of answers
(p > 0.05).
Respondents’ perceived quality of instructions, difficulty/ease in the comprehension of
questions and difficulty/ease in making choices of answers are not significantly influenced
by spoken language, education and race of respondents. Evidence supporting the
appropriateness of the maternal and child health questionnaire (core) is robust but this is
not to underrate the need for improvement in some aspects of the questionnaire.
1
Table of Contents
EXECUTIVE SUMMARY .............................................................................................................................. 1
1.0
PREAMBLE ...................................................................................................................................... 6
2.0
OBJECTIVE OF REPORT ................................................................................................................. 6
3.0
METHODOLOGY............................................................................................................................ 7
3.1
Preliminary Analysis of Qualitative Data ................................................................................ 7
3.2
Measurement of variables ....................................................................................................... 7
3.3
Analysis of Quantitative Data .................................................................................................. 8
3.4
Processing of Selected Codes ................................................................................................. 8
4.0
RESULTS ......................................................................................................................................... 8
4.1
Profile of Respondents ............................................................................................................. 8
Table 1: Socio-demographic characteristics of respondents (N =10) ................................................. 9
4.2
Primary Findings from the Qualitative Analysis .................................................................... 9
Table 2: Codes that Emergent from Data ............................................................................................. 10
4.3
Univariate Analysis of Perceived Quality of Instructions, Comprehension of Questions
and Making Choices of Answers among Respondents ................................................................... 10
4.3.1
Perceived Quality of Instructions...................................................................................... 10
Figure 1: A Venn diagram showing the univariate distribution of respondents’ perceived quality
of instructions ........................................................................................................................................... 11
4.3.2.
Comprehension of questions ....................................................................................... 11
Figure 2: A Venn diagram showing the univariate distribution of respondents’ difficulty/ease in
comprehending questions ...................................................................................................................... 12
4.3.3.
Making choices of answers ........................................................................................... 12
Figure 3: A Venn diagram showing the univariate distribution of respondents’ ease/difficulty in
making choices of answers ..................................................................................................................... 13
4.4
4.4.1
Bivariate Analysis of Data ...................................................................................................... 13
Spoken language and quality of instructions ................................................................ 13
Figure 4: A stacked bar chart showing the cross distribution of spoken language and
respondents’ perceived quality of instructions .................................................................................... 14
4.4.2
Spoken language and comprehension of questions ..................................................... 14
2
Figure 5: A stacked bar chart showing the cross distribution of spoken language and the
respondents’ difficulty/ease in comprehending questions ................................................................ 15
4.4.3
Spoken language and making choices of answers ....................................................... 15
Figure 6: A stacked bar chart showing the cross distribution of spoken language and
respondents’ difficulty/ease in making choices of answers ............................................................... 16
4.4.4
Education and perceived quality of instructions ............................................................ 16
Figure 7: A stacked bar chart showing the cross distribution of highest educational qualification
and respondents’ perceived quality of instructions ............................................................................ 17
4.4.5
Education and comprehension of questions .................................................................. 17
Figure 8: A stacked bar chart showing the cross distribution of highest educational qualification
and respondents’ difficulty/ease in comprehending questions ......................................................... 18
4.4.6
Education and making choices of answers ..................................................................... 18
Figure 9: A stacked bar chart showing the cross distribution of highest educational qualification
and respondents’ difficulty/ease in making choices of answers ........................................................ 19
4.4.7
Race and perceived quality of instructions .................................................................... 19
Figure 10: A stacked bar chart showing the cross distribution of race and respondents’ perceived
quality of instructions .............................................................................................................................. 20
4.4.8
Race and comprehension of questions .......................................................................... 20
Figure 11: A stacked bar chart showing the cross distribution of race and respondents’
difficulty/ease in comprehending questions ........................................................................................ 21
4.4.9
Race and making choices of answers ............................................................................. 21
Figure 12: A stacked bar chart showing the cross distribution of race and respondents’
difficulty/ease in making choices of answers ....................................................................................... 22
5.0
Graphical representation of the Contents of Three Emergent Codes: Ambiguity of
instructions, Difficulty in Understanding Questions and Difficulty in Choosing Answers .............. 22
5.1
Graphical representation of the code of ambiguity of instructions ................................. 22
Figure 13: Graphic Representation of Code on Ambiguity of instructions ....................................... 24
5.2
Difficulty in understanding questions .................................................................................. 24
Figure 14: Graphical representation of the code of difficulty in understanding questions ............ 26
5.3
Difficulty in choosing answers .............................................................................................. 26
Figure 15: Graphical representation of the code of difficulty in choosing answers ........................ 27
3
6.0 Distribution of respondents’ activities/conditions before they got pregnant, healthcare visits
in the 12 months before they got pregnant with their new baby, healthcare provider’s
management of respondents’ high blood pressure during their most recent pregnancy and
items that respondents’ new baby have been placed to sleep with, in the 2 weeks before the
survey......................................................................................................................................................... 27
6.1
Respondents’ activities/conditions before they got pregnant ......................................... 28
Figure 16: Distributions of activities/conditions of respondents before they got pregnant
(Question 2) .............................................................................................................................................. 28
6.2
Respondents’ healthcare visits in the 12 months before they got pregnant with their
new baby ............................................................................................................................................... 28
Figure 17: Healthcare visits in the 12 months before respondents got pregnant with their new
baby (Question 4) .................................................................................................................................... 29
6.3
Healthcare provider’s management of respondents’ high blood pressure during their
most recent pregnancy ....................................................................................................................... 29
Figure 18: Healthcare provider’s management of respondents’ high blood pressure during their
most recent pregnancy (Question 16) ................................................................................................... 30
6.4
Items that respondents’ new baby have been placed to sleep with, in the 2 weeks
before the survey ................................................................................................................................. 30
Figure 19: Items that respondents’ new baby have been placed to sleep with in the 2 weeks
before the survey (Question 40) ............................................................................................................ 31
Core Testing Demographics ....................................................................................................................... 32
Results ........................................................................................................................................................ 33
Question Analysis ................................................................................................................................... 33
Core Testing Question-by-Question Detailed Analysis ........................................................................... 34
Core Testing Question Abbreviated Analysis ......................................................................................... 53
7.0
CONCLUSIONS ............................................................................................................................ 54
8.0 RECOMMENDATIONS ....................................................................................................................... 54
Appendix 1: Case summaries of all respondents (N = 10) .......................................................................... 57
Appendix 2: The Code of Ambiguity of Instructions ................................................................................... 58
Appendix 3: The Code of Clarity of Instructions ......................................................................................... 61
Appendix 4: The Code of Difficulty in Understanding Questions ............................................................... 63
Appendix 5: The Code of Ease in Understanding Questions ...................................................................... 65
Appendix 6: The Code of Difficulty in Choosing Answers ........................................................................... 67
4
Appendix 7: The Code of Ease in Choosing Answers .................................................................................. 69
Appendix 8: The Code of Recommendations Regarding Ambiguous Instructions ..................................... 71
Appendix 9: The Code of Recommendations Regarding Difficulty in Understanding Questions ............... 74
Appendix 10: The Code of Recommendations Regarding Difficulty in Choosing Answers ........................ 76
Appendix 11: The Code of Comments or Suggestions for Improving the Survey....................................... 77
5
1.0
PREAMBLE
Cognitive and field-testing is sine qua non in developing standard measurement tools.
This report of the cognitive testing of the core version of the child and maternal health
questionnaire highlights respondents’ understanding and interpretations of, as well as
reactions towards the questionnaire. The field-testing is more or less an actual study
revealing real-life performance of the questionnaire. The field test was conducted among
ten English or Spanish speaking mothers. On completing the questionnaire, respondents’
opinions and experiences were explored, leading to production of textual data. This report
is essentially the presentation of the analyses of the synopsis of this textual data.
2.0
OBJECTIVE OF REPORT
The general objective of this report is to present respondents’ experiences in answering
the questionnaires during the field-testing.
The specific objectives are to:
1. Describe
the
distributions
of
the
socio-demographic
characteristics
of
respondents.
2. Highlight emergent codes from data.
3. Highlight the distribution of respondents in terms of their opinions regarding
quality of instructions, difficulty/ease in the comprehension of questions and
difficulty/ease in making choices of answers.
4. Examine the association and the significance of association between respondents’
spoken language, education, and race on one hand, and perceived quality of
instructions, difficulty/ease in the comprehension of questions and difficulty/ease
in making choices of answers on the other hand.
5. Graphically represent the codes of ambiguity of instructions, difficulty in
understanding questions and difficulty in choosing answers.
6. Highlight
the
distribution
of
actual
data
concerning
respondents’
activities/conditions before they got pregnant, healthcare visits in the 12 months
before they got pregnant with their new baby, healthcare provider’s management
of respondents’ high blood pressure during their most recent pregnancy and items
that respondents’ new baby have been placed to sleep with, in the 2 weeks before
the survey.
7. Conclude and Highlight recommendations for improving the questionnaire.
6
3.0
METHODOLOGY
The systematic process was mixed and multi-phased. This is described as follows:
3.1
Preliminary Analysis of Qualitative Data
The summary of respondents’ expressed experiences during field-testing of the
questionnaire was subjected to content analysis. A pseudo name was assigned to each of
the ten respondents. The summary featured the responses to four questions: if the
questionnaire items were hard or easy to understand, if respondents had any trouble
choosing between the answer choices, if the instructions were clear and if respondents
had any other comments or suggestions for improving the survey. Interviewers recorded
responses and also asked for recommendations regarding the first three questions, if any.
In other words, quality of instructions, comprehension of questions, making choices of
answers (three variables) were among the variables of the post-field testing.
Recommendations to improve these three issues (additional three variables) were
variables of this study while general comments or suggestions for improving the survey
was also another variable. The summary of textual data were prepared for analysis by
gathering each respondent’s contributions in separate word documents. Each document
was named and saved using each respondent’s socio-demographic and other
characteristics, including the pseudo name that was assigned to each respondent. Nvivo
10 for windows was used for the content analysis.
3.2
Measurement of variables
Measurement of variables were attempted, using some of the codes that emerged from
data. Ten codes (six convergent and four divergent) emerged from the data at the end of
the preliminary analysis. Measurement included assigning numbers to data in a
meaningful way. The nominal level of measurement was adopted by identifying
respondents who populated each of the 3 pairs of the convergent codes and assigning
numbers 1 and 3 to them. Any respondent who happens to feature in a relevant pair of
the codes was thereafter stripped of assigned number 1 and 3, and then assigned number
2. For example, Erica was reported to have stated that “instructions were generally
understandable with some mild confusion experienced with instructions that offered
arrows next to Yes/No answer choices”. As a result, Erica was featured in the codes ‘clarity
of instructionss’ and ‘ambiguity of instructionss’. During measurement, Erica was assigned
1 and 3. Because she populated the two codes/belonged to the two groupings, she
automatically ceased to belong to the two groups but a middle level, ambivalent group.
So, while some respondents like Esmeralda was assigned 1 because she only belonged to
7
the ‘clarity of instructionss’ group/code, and while Bree was assigned 3 because she only
belonged to the ‘ambiguity of instructionss’ group/code, Erica was assigned 2 because
she neither belonged to the former nor the latter group/code. The process that was
applied in the measurement of quality of instruction was repeated in the measurements
of comprehension of questions and making choices of answers.
3.3
Analysis of Quantitative Data
Simple percentile analysis was used to assess the univariate distributions of data, which
were visualized using Venn diagrams. Bivariate analysis, i.e. cross-distributions of data
were examined using stacked bar charts. Chi-square was used to test the significance of
associations between the levels of two pairs of variables. Quantitative data analyses were
done using Statistical Package for Social Sciences (version 24). Data visualization were
accomplished with Microsoft excel, Microsoft Word SmartArt, Photopea and Venn
Diagram Maker.
3.4
Processing of Selected Codes
The summary of three codes including the codes of ambiguity of instructions, difficulty in
understanding questions and difficulty in choosing answers were processed. Participants’
submissions in these codes were edited minimally without changing the message of the
respondent, to bring out the issues of concern in the codes. The codes were later
represented graphically.
4.0
RESULTS
4.1
Profile of Respondents
The mean±SD of age was 30.67±5.94 (range = 22-38 years). The age of seven of the ten
respondents was between 26 to 45 while two of the ten respondents aged between 18 to
25 years. These indicates that the respondents were generally not young mothers. One
respondent did not state her age. One in two respondents (50%) completed high school,
while four of ten respondents achieved greater than high school education. This reflects
that basic education is optimal among the respondents. Hispanics made up half of the
respondents, Black Non-Hispanics were three of ten whereas just one respondent was
White Non-Hispanic. One respondent did not indicate her race. Respondents were evenly
distributed in terms of the two spoken languages— English and Spanish. Majority of the
respondents earned from $25,001 to $50,000 while one respondent did not indicate her
income. The summary of the socio-demographic characteristics of respondents is
8
presented in Table 1 while the case by case details of the same is presented in appendix
1.
Table 1: Socio-demographic characteristics of respondents (N =10)
Socio-demographic
characteristic
Age*
Sub-groups
Frequency
Percentage (%)
18-25
26-35
36-45
Missing
2
4
3
1
20.0
40.0
30.0
10.0
Education
Less than High
School
High School
Greater than High
School
1
10.0
5
4
50.0
40.0
Race
Black Non-Hispanic
White Non-Hispanic
Hispanic
Missing
3
1
5
1
30.0
10.0
50.0
10.0
Spoken Language
English Speakers
Spanish Speakers
5
5
50.0
50.0
Household Income
Less than $25,000
2
$25,001 to $50,000
4
$50,001 to $85,000
1
$85,001 and above
2
Missing
1
*The mean ±SD of age was 30.67± 5.94, minimum= 22, maximum= 38.
4.2
20.0
40.0
10.0
20.0
10.0
Primary Findings from the Qualitative Analysis
The content analysis of qualitative data yielded ten codes, six convergent codes and four
divergent. The six convergent codes were populated by respondents’ answers that align
with the code titles. For instance, the code of “difficulty in understanding questions” is like a
unit containing responses of respondents who opined that questions were difficult to
understand. Contrarily, the four divergent codes contains responses that reflect the title
of the codes and are multifarious at the same time. For example, the code ‘comments or
suggestions for improving the survey’ contains all responses about comments or
9
suggestions. The characteristics of these codes are summarized in Table 2 while all codes
and their contents are presented in appendixes 2 to 11.
Table 2: Codes that Emergent from Data
s/no
Code title
Type of code
1
2
3
4
5
6
7
Difficulty in understanding questions
Ease in understanding questions
Difficulty in choosing answers
Ease in choosing answers
Clarity of instructions
Ambiguity of instructions
Recommendation regarding ambiguous
instructions
Recommendation regarding difficulty in choosing
answers
Recommendation regarding difficulty in
understanding questions
Comments or suggestions for improving the
survey
Convergent
Convergent
Convergent
Convergent
Convergent
Convergent
Divergent
Number of
respondents that
populated the code
7
9
4
10
7
9
8
Divergent
3
Divergent
5
Divergent
10
8
9
10
4.3
Univariate Analysis of Perceived Quality of Instructions,
Comprehension of Questions and Making Choices of Answers among
Respondents
4.3.1 Perceived Quality of Instructions
Only one of the ten respondents was exclusive in reporting clarity of questionnaire
instructions. Three of ten respondents were also exclusive in reporting that instructions
were ambiguous. Nevertheless, the preponderance of respondents (6, 60.0%) were
ambivalent. Majority of respondents (70.0%) expressed clarity of instructions but also
pointed out limitations of this clarity. Hence, 70.0% of respondents, as visualized in Figure
1, acknowledged the clarity of questionnaire instructions.
10
Figure 1: A Venn diagram showing the univariate distribution of respondents’ perceived quality of
instructions
4.3.2. Comprehension of questions
An overwhelming majority of respondents (9, 90.0%) reported that they had no difficulty
in comprehending the questions. A sizeable chunk of this majority (6, 60.0%) were
ambivalent, by noting that their comprehension was limited. Notwithstanding, three
(30.0%) respondents were exclusive in expressing the comprehensibleness of the
questionnaire. The comprehensibleness of the questionnaire is evident but also limited,
as highlighted in Figure 2.
11
Figure 2: A Venn diagram showing the univariate distribution of respondents’ difficulty/ease in
comprehending questions
4.3.3. Making choices of answers
Respondents were unanimous in proclaiming their ease in choosing answers. However,
four (40.0%) further reported limitations to the ease of choosing answers. This distribution,
represented in Figure 3, shows that making choices of answers was the least challenging
of the three dependent variables whose univariate distributions have been examined so
far.
12
Figure 3: A Venn diagram showing the univariate distribution of respondents’ ease/difficulty in
making choices of answers
4.4
Bivariate Analysis of Data
In bivariate analysis of data, descriptive cross-distribution, association and significance of
association between independent and dependent variables were attempted. The
independent variables were respondents’ spoken language, education and race while the
dependent variables were perceived quality of instructions, comprehension of questions
and making choices of answers. Therefore, the general null hypotheses reads as follows:
there is no significant association between respondents’ spoken language, education, race
and perceived quality of instructions, comprehension of questions and making choices of
answers. The report of bivariate analyses are presented in the sub-sections below:
4.4.1
Spoken language and quality of instructions
The only respondent that was exclusive in describing clarity of instructions was a Spanish
speaker. Still, two Spanish speaking respondents exclusively reported ambiguity of
instructions, only one English speaking respondent reported this ambiguity. Meanwhile,
four of the five English-speaking respondents were ambivalent. This faintly demonstrates
that being English speaking ordinarily predisposes towards the reportage of ambiguity of
instructions but this is not statistically significant (chi-square = 2.000, p > 0.05). Therefore,
the null hypothesis is accepted, spoken language is not significantly associated with
perceived quality of instructions. This means that perceived quality cannot be explained
13
in terms of respondents’ spoken language. The cross distribution of spoken language and
perceived quality of instructions is represented in Figure 4.
100%
90%
80%
70%
Percentage
60%
50%
40%
30%
20%
10%
0%
English
Spanish
Ambiguity of Instructions
20.0%
40.0%
Ambivalence of Instructions
80.0%
40.0%
Clarity of Instructions
0.0%
20.0%
Chi-square = 2.000 (p = 0.368), N = 10.
Figure 4: A stacked bar chart showing the cross distribution of spoken language and respondents’
perceived quality of instructions
4.4.2 Spoken language and comprehension of questions
Ambivalence in the comprehension of questions was equally common to English and
Spanish speakers because 60.0% of each group’s members were ambivalent. While one
English speaker was exclusive in expressing her ease in comprehending questionnaire
questions, two Spanish speakers reported this ease. The only respondent who reported
exclusive difficulty in comprehending questions was English speaking. This descriptive
data represented in Figure 5 suggests that Spanish speakers are a bit more favourably
placed in terms of comprehension of questions but this is not significantly so (chi-square
= 1.333, p > 0.05). Therefore, the null hypothesis is accepted, spoken language is not
significantly associated with comprehension of questions. Comprehension of questions is
independent of spoken language.
14
100%
90%
80%
70%
Percentage
60%
50%
40%
30%
20%
10%
0%
English
Spanish
Ease in Understanding Questions
20.0%
40.0%
Ambivalence in Understanding
Questions
60.0%
60.0%
Difficulty in Understanding
Questions
20.0%
0.00%
Chi-square = 1.333 (p = 0.513), N = 10.
Figure 5: A stacked bar chart showing the cross distribution of spoken language and the
respondents’ difficulty/ease in comprehending questions
4.4.3
Spoken language and making choices of answers
English and Spanish speaking respondents were equally distributed in terms of ease and
ambivalence in making choices of answers. Incidentally, more respondents (60.0%)
exclusively reported ease rather than ambivalence in choosing answers which was
reported by 40.0% of respondents in the two spoken language categories. These are
represented in Figure 6. The chi-square of this analysis is 0.000 (p > 0.05). Therefore, the
null hypothesis is accepted, spoken language is not significantly associated with
difficulty/ease in making choices of answers. Making choices of answers is independent
of spoken language.
15
100%
90%
80%
70%
Percentage
60%
50%
40%
30%
20%
10%
0%
English
Spanish
Ease in choosing answers
60.0%
60.0%
Ambivalence in choosing answers
40.0%
40.0%
Chi-square = 0.000 (p = 1.000), N = 10.
Figure 6: A stacked bar chart showing the cross distribution of spoken language and respondents’
difficulty/ease in making choices of answers
4.4.4 Education and perceived quality of instructions
The only respondent who completed less than high school education was exclusive in
reporting ambiguity of questionnaire instructions. All the four respondents who had
greater than high school education maintained ambivalent attitude in their perceived
quality of instructions. Among the five respondents who completed high school, one
respondent was exclusive in reporting clarity of instructions. The chi-square of this analysis
was = 6.000 (p > 0.05). Therefore, the null hypothesis is accepted, education is
independent of perceived quality of instructions.
16
100%
90%
80%
70%
Percentage
60%
50%
40%
30%
20%
10%
0%
Less than High
School
High School
Greater than High
School
100.0%
40.0%
0.0%
Ambivalence of Instructions
0.0%
40.0%
100.0%
Clarity of Instructions
0.0%
20.0%
0.0%
Ambiguity of Instructions
Chi-square = 6.000 (p =0.199), N = 10.
Figure 7: A stacked bar chart showing the cross distribution of highest educational qualification
and respondents’ perceived quality of instructions
4.4.5 Education and comprehension of questions
The only ‘less than high school’ respondent was exclusive in reporting ease in
understanding questions. Only 1 of 5 (20.0%) of ‘high school’ and ‘greater than high
school’ (25.0%) respondents were exclusive in reporting ease in understanding questions.
The only respondent that was exclusive in reporting difficulty in understanding questions
had ‘greater than high school’ education. These descriptive findings, represented in Figure
8, shows no clear-cut influence of education on comprehension and was not statistically
significant (chi-square = 4.333, p > 0.05). Therefore, the null hypothesis is accepted,
education is not associated with comprehension of questions.
17
100%
90%
80%
70%
Percentage
60%
50%
40%
30%
20%
10%
0%
Less than High
School
High School
Greater than High
School
100.0%
20.0%
25.0%
Ambivalence in Understanding
Questions
0.0%
80.0%
50.0%
Difficulty in Understanding
Questions
0.0%
0.0%
25.0%
Ease in understanding Questions
Chi-square = 4.333 (p = 0.362), N = 10.
Figure 8: A stacked bar chart showing the cross distribution of highest educational qualification
and respondents’ difficulty/ease in comprehending questions
4.4.6 Education and making choices of answers
The only ‘less than high school’ respondent was exclusive in reporting ease in choosing
answers. Exclusive ease in choosing answers was also reported by 3 of the 5 (60.0%) and
2 of the 4 (50.0%) of ‘high school’ and ‘greater than high school’ respondents respectively.
No respondent was exclusive in reporting difficulty in choosing answers. The descriptive
findings depicted in Figure 9 are insignificantly associated with each other (chi-square =
0.833, p > 0.05). Therefore, the null hypothesis is accepted, education is not significantly
associated with making choices of answers.
18
100%
90%
80%
70%
Percentage
60%
50%
40%
30%
20%
10%
0%
Ease in choosing answers
Ambivalence in choosing answers
Less than High
School
High School
Greater than High
School
100.0%
60.0%
50.0%
0.0%
40.0%
50.0%
Chi-square = 0.833 (p = 0.659), N = 10.
Figure 9: A stacked bar chart showing the cross distribution of highest educational qualification
and respondents’ difficulty/ease in making choices of answers
4.4.7
Race and perceived quality of instructions
Two Black respondents (66.7%), the only White respondent (100.0%) and 3 (60.0%)
Hispanic respondents reported ambivalence of instructions. The only respondent who was
exclusive in reporting clarity of instructions was Hispanic. The chi-square of this analysis
is 1.400 (p > 0.05). Therefore, the null hypothesis is accepted, race is not significantly
associated with perceived quality of instructions. The cross distribution of race and the
respondents’ perceived quality of instructions is represented in Figure 10.
19
100%
90%
80%
70%
Percentage
60%
50%
40%
30%
20%
10%
0%
Black Non-Hispanic
White Non-Hispanic
Hispanic
Ambiguity of Instructions
33.3%
0.0%
20.0%
Ambivalence of Instructions
66.7%
100.0%
60.0%
Clarity of Instructions
0.0%
0.0%
20.0%
Chi-square = 1.400 (p = 0.844), N = 10.
Figure 10: A stacked bar chart showing the cross distribution of race and respondents’ perceived
quality of instructions
4.4.8
Race and comprehension of questions
The only White respondent (100.0%) reported ambivalence in understanding questions
whereas 4 of the 5 Hispanics (80.0%) reported this ambivalence 1 of the 3 Blacks (33.3%)
did. Of the two respondents who were exclusive in reporting ease in understanding
questions, 1 (33.3%) was Black while the other was (20.0%) was Hispanic. Chi-square
analysis (3.200, p > 0.05) shows that comprehension of questions is not significantly
associated with race. Therefore, the null hypothesis is accepted. Comprehension is
independent of race. The cross distribution of race and the respondents’ comprehension
of questions is represented in Figure 11.
20
100%
90%
80%
70%
Percentage
60%
50%
40%
30%
20%
10%
0%
Black NonHispanic
White NonHispanic
Hispanic
Ease in understanding Questions
33.3%
0.0%
20.0%
Ambivalence in Understanding
Questions
33.3%
100.0%
80.0%
Difficulty in Understanding
Questions
33.3%
0.0%
0.0%
Chi-square = 3.200 (p = 4.000), N = 10.
Figure 11: A stacked bar chart showing the cross distribution of race and respondents’
difficulty/ease in comprehending questions
4.4.9
Race and making choices of answers
All the three Black (100.0%) and two (40.0%) Hispanic respondents reported exclusive ease
in making choices of answers. The only White (100.0%) and 3 (60.0%) of Hispanic
respondents reported ambivalence in choosing answers. Chi-square analysis (4.140, p >
0.05) shows that race is not significantly associated with the making of choices of answers.
Therefore, the null hypothesis is accepted. Race is independent of difficulty/ease in
making choice of answers. The cross distribution of race and difficulty/ease in making
choices of answers is represented in Figure 12.
21
100%
90%
80%
70%
Percentage
60%
50%
40%
30%
20%
10%
0%
Ease in choosing answers
Ambivalence in choosing answers
Black NonHispanic
White NonHispanic
Hispanic
100.0%
0.0%
40.0%
0.0%
100.0%
60.0%
Chi-square = 4.140 (p = 0.126), N = 10.
Figure 12: A stacked bar chart showing the cross distribution of race and respondents’
difficulty/ease in making choices of answers
5.0
Graphical representation of the Contents of Three Emergent
Codes: Ambiguity of instructions, Difficulty in Understanding
Questions and Difficulty in Choosing Answers
5.1
Graphical representation of the code of ambiguity of instructions
The details of some respondent’s experiences regarding ambiguity of instructions are
represented in Figure 13. Respondents’ demographic and other characteristics are
presented like signboard to their expressions.
22
Andrea, 29, Hispanic, Spanish speaking, completed high
school, BFA-MA-005
•Andrea missed the instruction box between Q27 and Q28. She noted that she did not notice or read it. For Q36,
Q39, Q40, Q53, and Q55, Andrea noted she understood the questions to mean she was only to respond the one
way in which the question applied to her. She wanted to just answer Yes for the one way, she puts her baby to
sleep while answering Q36, Q39 and Q40 and Q53. She was overwhelmed by the amount of questions and
options.
•Interviewer’s observation: Survey fatigue was also noted as a factor with Q36, Q39, Q40, Q53, and Q55, she was
becoming tired of the questions and of having to mark each individual boxes.
Bailey, 31 years old, Black non-hispanic, English
speaking, completed greater than high school, BFA-MI005
•Bailey also missed the instruction box between Q27 and Q28, causing her to answer Q28 which did not apply to
her.
Bella, 40 years old, Black Non-hispanic, English
speaking, completed greater than high school, BFA-MI002
•Bella also missed seeing the instruction boxes between Q27 and Q28. Rather than skipping Q28 to proceed to
Q29, since she had not consumed any alcohol during her pregnancy, she answered Q28. Similar experience
occurred between Q42 and Q43.
Bree, 22 years old, Black non-hispanic, English speaking,
completed high school, BFA-MI-003
•Bree experienced difficulty repeatedly with instruction boxes throughout survey. She missed seeing instruction
boxes between Q4 and Q5, as well as Q27 and Q28. Bree also experienced difficulty with Q44, noting she has not
yet had her postpartum checkup due to giving birth only recently, however an appointment is scheduled for this
checkup in the coming weeks. In reading this question she wondered whether she should include this
information and then missed the instruction box with an arrow directing her to go to Q46 if she answered No.
She noted it would have been helpful to have this instruction box closer to the No, or worded differently. She
also experienced this difficulty with instruction box in Q17.
Catalina, 22, Hispanic, Spanish speaking, completed less
than high school, BFA-MA-001
•Catalina experienced some difficulty with instruction boxes placed between Q4 and Q5. She noted that she
noticed and read the instruction box but she did not fully comprehend instruction and therefore proceeded to
the next question even though due to her answers to Q4, she should have proceeded to Q6. She reported the
instructions that included an arrow with instruction to go to a specific question were more helpful and easier to
understand than the instruction boxes.
•Catalina also experienced difficulty with Q27 as she did not proceed to answer No for each sub-question, but
rather circled No at the top of the column. She noted she simply answered No to the main question because she
did not drink alcohol at all throughout her pregnancy and skipped reading the rest of the options in order to
proceed to next question.
23
Erica, 34 years old, Hispanic, English speaking,
completed greater than high school, BFA-MI-001
•Erica experienced some mild confusion with instructions that offered arrows next to Yes/No answer choices.
Erica expressed that arrows pointing away from the No provide clear direction on where to go next, because it
points to an instruction box that states - Go to Question X. However, the arrow pointing away from the Yes
answer choice box does not have a similar instruction box, therefore she was left to wonder whether she was to
keep going and whether she was to keep answering all questions that followed or only some questions. She
assumed she was being directed to the very next question but with a sense of doubt about her decision to keep
moving forward in this manner.
•Erica also noted difficulty with instruction box on page 4, before, Q16 -If you had high blood pressure before or
during your pregnancy, go to Question Core 16, if not, go to Question Core 17. Respondent stated she had to reread instruction several times to determine whether it was asking her to proceed to Question 16 only if she had
high blood pressure during one of the two time periods (before or during pregnancy) or if she had high blood
pressure during one of the two as well as both time periods.
Juanita, 28 years old, Hispanic, Spanish speaking,
completed high school, BFA-MA-004
•Juanita missed the instruction box between Q15 and Q16, noting that she did not visually notice them and
therefore did not read instructions. This occurred again with the instruction box between Q27 and Q28. Juanita
She also experienced difficulty with instructions on Q33, proceeding to Page 8 and not answering the next set of
questions on that page. She noted that at this point, survey fatigue was becoming significant and she read the
instruction box rather quickly, only noticing the beginning of the instruction. For Q39 and Q40, the wording of
the questions and lack of instruction caused respondent to interpret that she was only to answer Yes to the
option that applied to her, and therefore she left all No check boxes blank. For Q41 her answer was No to this
question, however she failed to mark it. Juanita followed instructions but incorrectly. She thought the arrow
pointing toward the instruction box instructing her to go to Q43 was meant for her answer of No. Therefore, she
skipped Q42 and went to Q43, but left it blank because the question did not apply to her. She also noted survey
fatigue was a factor here.
Luca, 36 years old, Spanish speaking, completed high school,
BFA-MA-002
•Luca experienced difficulty throughout the survey with instruction boxes between questions, as well as Yes/No
instructions with arrows pointing out to instruction boxes. Luca exhibited mixed experience, fully understanding
instructions for some questions and other times experiencing confusion. Difficulty occurred with instruction
boxes between Q4 and Q5, Q15 and Q16, Q19 Y/N instruction box with arrow, Q23 Y/N instruction box with
arrow.
Meredith, 38 years old, White non-hispanic, English, Greater
than high school, $85000, Mail, BFA-MI-004
•Meredith noted that she did not notice the section headings Before / During / After Pregnancy. She suggested
making the whole heading of the page a defined section rather than the column alone as a way to make this
more visually clear for respondents.
Figure 13: Graphic Representation of Code on Ambiguity of instructions
5.2
Difficulty in understanding questions
Experiences of difficulties in understanding questions are represented in Figure 14. Details
of respondents are presented along the same.
24
Andrea, 29 years old, Hispanic, Spanish speaking,
completed high school, BFA-MA-005
•She did experience some confusion with a couple questions.
•Andrea noted that for Q18, she was not sure what was meant by “las señales de advertencia” expressing
that she did not connect Q18 to Q17 and therefore did not understand that the warning signs being asked
about in Q18 were the same one which were defined in Q17. For Q32, she stated that she was unsure if
the question was asking whether her baby stayed in the hospital extra days without her (she thought
about babies that go to NICU) or if the question referred to the time period they were both in the hospital
together.
Bella, 40 years old, Black Non-hispanic, English
speaking, completed greater than high school, BFAMI-002
•Bella reported experiencing some confusion with Questions 36-40 which asked about baby’s sleeping in
the past 2 weeks. She noted due to Q32 and instruction box - If your baby is still in the hospital, go to
Question Core 41, she wondered whether she was supposed to think about the first 2 weeks after bringing
baby home from hospital or if she was supposed to think about the most recent 2 weeks. Note that
respondent’s baby is now 10 months old. Respondent noted she decided to answer the question thinking
about the 2 weeks prior to the current date.
Bree, 22 years old, Black non-hispanic, English
speaking, completed high school, BFA-MI-003
•Bree reported having difficulty with Q11, noting that the Ask me… section of the question caused
confusion about how to answer the question. The framing of the question in the first person Ask me… if I
was drinking alcohol caused respondent to question whether she was supposed to answer No because she
did not consumealcohol during her pregnancy, or Yes because her doctor had spoken with her about this
topic. She experienced similar confusion for sub-questions g, h, i, k, and l.
Erica, 34 years old, Hispanic, English speaking,
completed greater than high school, BFA-MI-001
•Erica stated that she had to re-read some questions a few times to ensure that she was thinking of the
correct time frame i.e., 12 months before you got pregnant, before or during your pregnancy, last 3
months of your pregnancy. Respondent also noted experiencing some confusion and difficulty with fully
understanding what was meant by postpartum care in Q53.
Esmeralda, 36 years old, Hispanic, Spanish speaking,
completed high school, BFA-MA-003
•Esmeralda reported being confused with Q3-a and stated that she had to re-read several times to ensure
she was interpreting correctly— whether the question was asking about type 1 or type 2 diabetes, or
gestational diabetes.
25
Juanita, 28 years old, Hispanic, Spanish speaking,
completed high school, BFA-MA-004
•For Q56, Juanita did not understand the meaning of “ingreso total”. For Q57, while she understood the question
fully, she did not understand what was being referred to when it stated “dependian de este dinero” because Q56
and Q57 were not connected in her mind. Q6, Q7, and Q8 were identified as repetitive. Juanita also expressed
that the survey had too many questions, and she began to experience a desire to finish as quickly as possible
midway through the survey.
Meredith, 38 years old, White non-hispanic, English,
Greater than high school, $85000, Mail, BFA-MI-004
•Meredith noted that Q13 caused some minor confusion related to the Covid19 vaccination. She received two
doses prior to pregnancy and a booster shot during pregnancy so she was unsure how to answer this question.
Figure 14: Graphical representation of the code of difficulty in understanding questions
5.3
Difficulty in choosing answers
Details of the emergent code of difficulty in choosing answers is represented in Figure
15. Demographic and other characteristics of respondents are positioned on top of their
contributions.
26
Andrea, 29, Hispanic, Spanish speaking, completed high
school, BFA-MA-005
•Andrea did not have an answer choice that applied to her situation for Q6, Q7, and Q8, therefore leaving these
blank. Andrea said that she did not have private health insurance, Medicaid, or another type of health insurance
but she did have health coverage through the hospital where she was receiving medical care. She described this
as a plan that she had to submit an application to and provide proof of income in order to qualify and receive
discounted rates.
Erica, 34 years old, Hispanic, English speaking,
completed greater than high school, BFA-MI-001
•Erica stated that the questions that offered Likert scale answer options were mildly difficult for her noting that
she finds the words Sometimes and Rarely to be almost the same. She suggested removing the rarely or replacing
it with something like hardly at all.
Juanita, 28 years old, Hispanic, Spanish speaking,
completed high school, BFA-MA-004
•For Q11, Juanita left several blank as she was unsure/could not remember if these topics were brought up by her
provider.
Meredith, 38 years old, White non-hispanic, English,
Greater than high school, $85000, Mail, BFA-MI-004
•Meredith noted there were two questions that were lacking answer choices that applied to her situation, given
that her baby is a 3-wk old newborn. Q41 and Q44 were identified. Meredith noted that she and her spouse
have not yet made a decision about birth control given that they are only in the early stages of the postpartum
period. Similarly, she has not attended a postpartum checkup yet, but has one scheduled. She shared that
answering No to Q44 felt as if she had perhaps “missed” this appointment and not because she was too early in
her postpartum period.
Figure 15: Graphical representation of the code of difficulty in choosing answers
6.0 Distribution of respondents’ activities/conditions before they got
pregnant, healthcare visits in the 12 months before they got pregnant
with their new baby, healthcare provider’s management of
respondents’ high blood pressure during their most recent pregnancy
and items that respondents’ new baby have been placed to sleep with,
in the 2 weeks before the survey.
This section is a presentation of respondents’ actual experiences. The field-testing was
positioned as a window of opportunity to examine data, as follows:
27
6.1
Respondents’ activities/conditions before they got pregnant
The distributions represented in Figure 16 shows that all respondents had no difficulty
running errands alone, bathing/dressing themselves, concentrating/remembering,
walking/climbing stairs and hearing. Only one (10.0%) respondent indicated that she had
difficulty seeing.
Have difficulty doing errands alone such as visiting a
doctor's office or shopping because of physical, mental or
emotional condition?
100.0%
Have difficulty dressing or bathing yourself?
100.0%
Have serious difficulty concentrating, remembering, or
making decisions because of physical, mental or emotional
condition?
100.0%
Have serious difficulty walking or climbing stairs?
100.0%
Have serious difficulty seeing, even wearing glasses, or are
you blind?
90.0%
Have serious difficulty hearing, or are you deaf?
10.0%
100.0%
0.0%
No
20.0%
40.0%
60.0%
80.0%
100.0%
Yes
Figure 16: Distributions of activities/conditions of respondents before they got pregnant (Question
2)
6.2
Respondents’ healthcare visits in the 12 months before they got
pregnant with their new baby
Only four of ten (40.0%) respondents had regular checkup with an OB/GYN, while 2
respondents had check up with their family doctor and used urgent care in the 12 months
before they got pregnant. These are represented in Figure 17.
28
Visit to have my teeth cleaned
90.0%
10.0%
Visit for depression or anxiety
90.0%
10.0%
Visit for family planning or to get birth control
90.0%
10.0%
Visit to urgent care or the emergency room
80.0%
Visit for an injury, illness, or chronic condition
77.8%
Regular checkup with an OB/GYN
22.2%
60.0%
Regular checkup with a family doctor
0.0%
20.0%
40.0%
80.0%
20.0%
No
40.0%
20.0%
60.0%
80.0%
100.0%
Yes
Figure 17: Healthcare visits in the 12 months before respondents got pregnant with their new baby
(Question 4)
6.3
Healthcare provider’s management of respondents’ high blood
pressure during their most recent pregnancy
All (100.0%) and eight (80.0%) respondents reported that their healthcare provider talked
to them about checking their blood pressure regularly after pregnancy and during
pregnancy respectively. Two of six (33.3%) reported that they were referred to another
health provider while four of six respondents (66.7%) acknowledged that their healthcare
provider talked to them about their risk of having hypertension after pregnancy. These
distributions are represented in Figure 18.
29
Talk to me about my risk for having blood pressure (chronic
hypertension) and heart disease after pregnancy
33.3%
66.7%
Talk to me about regularly checking my blood pressure
after pregnancy
100.0%
Talk to me about getting a healthy weight after pregnancy
40.0%
Tell me to regularly check my blood pressure during
pregnancy
60.0%
20.0%
80.0%
Refer me to a different healthcare provider
66.7%
0.0%
No
20.0%
40.0%
33.3%
60.0%
80.0%
100.0%
Yes
Figure 18: Healthcare provider’s management of respondents’ high blood pressure during their
most recent pregnancy (Question 16)
6.4
Items that respondents’ new baby have been placed to sleep with, in
the 2 weeks before the survey
Five of eight (62.5%) and three of eight respondents (37.5%) respondents acknowledged
that their babies slept in a sleeping sack/wearable blanket and swaddled blanket in the 2
weeks before the survey, respectively. The item category least used to place baby to sleep
in the two weeks before the survey were comforters/quilts/non-fitted sheets, which was
reported by one of eight (12.5%) of respondents. These are represented in Figure 19
below:
30
Crib bumper pads (mesh or non-mesh)
75.0%
25.0%
Soft toys, cushions, or pillows, including nursing pillows
75.0%
25.0%
Comforters, quilt, blankets, or non-fitted sheets
87.5%
In a swaddled blanket
62.5%
In a sleeping sack or wearable blanket
0.0%
12.5%
37.5%
No
20.0%
Yes
37.5%
62.5%
40.0%
60.0%
80.0%
100.0%
Figure 19: Items that respondents’ new baby have been placed to sleep with in the 2 weeks before
the survey (Question 40)
31
Core Testing Demographics
10 women participated in the core testing phase. Most participants were between 30-39 (50.0%).
Household incomes were mostly fairly distributed among the different categories, with most earning
between $25,001-$50,000 (44.4%). When reporting education, only 10% identified as “Less than High
School”, while the remaining participants have at least completed High School. Half of the participants
reported their ethnicity as Hispanic (50.0%), 30.0% as Non-Hispanic Black (NH-Black), 10.0% as NonHispanic White (NH-White), and 10.0% as Other. 50% of participants reported English as their primary
language, while the remaining 50% reported Spanish. 80.0% of the participants reported the age of their
youngest child as 3 months or less, while the remaining reported their child older than 3 months.
Table 1. Core Testing Demographics
Education
Greater than high school
High School
Less than High School
Household Income
Less than $25000
$25001-$50000
$50001-$85000
$85000+
Age
Under 30
30-39
40 and over
Race
Hispanic
NH-Black
NH-White
Other
Language
English
Spanish
Age of youngest child
0-3 months
3-6 months
9-12 months
32
Count
4
5
1
Count
2
4
1
2
Count
4
5
1
Count
5
3
1
1
Count
5
5
Count
8
1
1
Percentage
40.0%
50.0%
10.0%
Percentage
22.2%
44.4%
11.1%
22.2%
Percentage
40.0%
50.0%
10.0%
Percentage
50.0%
30.0%
10.0%
10.0%
Percentage
50.0%
50.0%
Percentage
80.0%
10.0%
10.0%
Results
Question Analysis
All survey participants were asked the following questions:
1. Were the questions hard or easy to understand?
2. Did you have any trouble choosing between the answer choices?
3. Were the instructions clear?
From these questions, analysis was conducted to determine if survey participants encountered one of
these issues more than the other, while considering the language of the. See Figure 1, where it can be
visually observed that the most common issue seen was difficulty with the instructions.
Figure 1. Most Common Issues
Analysis of Variance (ANOVA) (two-way) was used to determine if this effect was statistically
significant (See Table 2 for ANOVA results). It was found that the there is a statistically significant
difference between average amount of issues reported based off the issue type.
Table 2. ANOVA Results
term
Sum Sq
Df
F value
Pr(>F)
(Intercept)
22.431
1
7.074
0.019
issue_type
23.981
2
3.781
0.049
survey_language
Residuals
13.564
44.394
1
14
4.278
0.058
33
Furthermore, it was observed that a participant may report between .02 and 5.73 (on average 2.875) more
questions with the Issue #3 (issues with instructions) more than they would for Issue #2 (issues choosing
between answer choices). This was determined using a Tukey’s HSD test where the results were found
significant at the 𝛼 = .05 level (See Table 3 for Tukey HSD results)
Table 3. Tukey's HSD Results
term
diff
lwr
upr
p adj
Issue 2- Issue 1
-0.91667
-3.92512
2.091791
0.7106
Issue 3- Issue 1
1.958333
-0.55872
4.47539
0.140032
Issue 3- Issue 2
2.875
0.020927
5.729074
0.048253
Core Testing Question-by-Question Detailed Analysis
From the interviews, all respondents believed overall that all the questions were easy to understand, there
was low to no difficulty choosing between answer choices, however, on average there was moderate
difficulty experienced with understanding and following the instructions provided.
Some of the most discussed topics during the interviews that can be seen throughout the question-byquestion analysis are (in no particular order):
1. Instruction boxes telling participants to skip the next question if they answered yes/no, were
oftentimes missed.
2. Questions requiring yes/no responses for multiple sub-questions were oftentimes misunderstood.
3. Instruction boxes between questions informing participants to skip certain sections if certain
conditions applied, were oftentimes missed.
4. Translation errors (Spanish).
5. Participants expressed survey fatigue, typically by around question 35+.
30 questions (See Figure 2- Most Problematic Questions by Language) were directly mentioned during
the interviews with the participants. 18 of these 30 questions were mentioned two or more times by
different participants. This section of the report will discuss in detail the notes provided to the
interviewers for each one of these 18 questions, beginning with the most reported questions in descending
order. The next section Core Testing Question Abbreviated Analysis will cover the remaining 12
questions in an abbreviated format.
34
Figure 2. Most Problematic Questions by Language
35
Question 27— Mentioned 6 times (3x English 3x Spanish)
Figure 3. Q27: English Mail
Discussion:
Participants oftentimes would visually miss the instruction box between Q27 and Q28, and rather than
skipping Q28 to proceed to Q29, they would proceed to answer Q28.
Recommendation:
Change position of instruction box between Q27 and Q28 to top of column or within question box.
36
Question 28— Mentioned 5 times (3x English 2x Spanish)
Discussion:
See discussion for Question 27.
Recommendation:
See recommendation for Question 27.
37
Question 16— Mentioned 3 times (3x English 3x Spanish)
Figure 4. Q16: English Mail
Discussion:
Participants oftentimes would visually miss the instruction box between Q15 and Q16.
Additionally, one participant also had trouble interpreting the instructions for this question— “If you had
high blood pressure before or during your pregnancy, go to Question Core 16, if not, go to Question Core
17.” The respondent stated she had to re-read instruction several times to determine whether it was asking
her to proceed to Question Core 16 only if she had high blood pressure during one of the two time periods
(before or during pregnancy) or if she had high blood pressure during one of the two as well as both time
periods.
Recommendation:
For the instruction box prior to Q16, add and/or to the instruction statement to clarify confusion between
before or during pregnancy - If you had high blood pressure before and/or during your pregnancy, go to
Question Core 16, if not, go to Question Core 17
For instruction boxes between questions, consider adding a bolder definition to the instruction box so that
it is more visually noticeable. Add bold letters, shadows, colors or add Please Read notices within
instruction boxes to make clearer an instruction will change the flow of answering questions.
38
Question 39— Mentioned 3 times (1x English 2x Spanish)
Figure 5. Q39: English Mail
Discussion:
Respondents noted they understood the question to mean they should select a single option for which way
they have placed their baby to sleep in the last two weeks, as opposed to answering yes/no for each single
option.
One respondent also felt confused based from the instruction box in Q32— “if your baby is still in the
hospital, go to Question 41”. She wondered whether she was supposed to think about the first 2 weeks
after bringing the baby home from the hospital or if she was supposed to think about the most recent 2
weeks.
Respondents also noted survey fatigue by the time they had reached this question.
Recommendation:
Consider making changes to the wording of this section in general. Remove the term new baby and
replace it with baby or infant. And replace new baby with newborn for questions that are pertaining to the
time period in the hospital after baby was born or right after coming home from hospital.
Consider an instruction box that instructs respondent to check Yes and No for each sub-question if it
applies to them, similar to the instructions included in Q3 and Q53.
Consider reducing the number of questions or breaking up the survey into two parts to address potential
survey fatigue.
39
Question 4— Mentioned 3 times (1x English 2x Spanish)
Figure 6. Q4: English Mail
Discussion:
Participants oftentimes would visually miss the instruction box between Q4 and Q5.
Recommendation:
Consider changing position of the instruction box to the top of Q4 so that it is visible prior to answering
the question.
40
Question 40— Mentioned 3 times (1x English 2x Spanish)
Figure 7. Q40: English Mail
Discussion:
Respondents noted they understood the question to mean they should select a single option from the subquestion, as opposed to answering yes/no for each single option.
Respondents also noted survey fatigue by the time they had reached this question.
Recommendation:
Consider an instruction box that instructs respondent to check Yes and No for each sub-question if it
applies to them, similar to the instructions included in Q3 and Q53.
Consider reducing the number of questions or breaking up the survey into two parts to address potential
survey fatigue.
41
Question 53— Mentioned 3 times (2x English 1x Spanish)
Figure 8.Q53: English Mail
Discussion:
Respondents noted they understood the question to mean they should select a single option from the subquestion, as opposed to answering yes/no for each single option.
One respondent also noted experiencing some confusion and difficulty with fully understanding what was
meant by postpartum care in Q53.
Respondents also noted survey fatigue by the time they had reached this question.
Recommendation:
Consider an instruction box that instructs respondent to check Yes and No for each sub-question if it
applies to them, similar to the instructions included in Q3 and Q53.
Consider defining postpartum care to include time period and type of postpartum care that is being
referred to in this question.
Consider reducing the number of questions or breaking up the survey into two parts to address potential
survey fatigue.
42
Question 11— Mentioned 2 times (1x English 1x Spanish)
Figure 9. Q11: English Mail
Discussion:
English Mail
Respondent felt confused while answering the “Ask me…” portion of the question. The use of firstperson here caused the respondent to question whether she was supposed to answer yes or no if one of the
boxes applied to her or if the doctor had asked her about it.
Spanish Mail
The respondent left several blank as she was unsure or could not remember if these topics were brought
up by her provider.
Recommendation
English Mail
Consider changing the wording to “Ask you…” instead.
Spanish Mail
Consider adding answer choices for “do not remember/recall”.
43
Question 15— Mentioned 2 times (2x Spanish)
Figure 10. Q15: English Mail
Discussion:
Participants oftentimes would visually miss the instruction box between Q15 and Q16.
Recommendation:
For instruction boxes between questions, consider adding a bolder definition to the instruction box so that
it is more visually noticeable. Add bold letters, shadows, colors or add Please Read notices within
instruction boxes to make clearer an instruction will change the flow of answering questions.
44
Question 17— Mentioned 2 times (1x English, 1x Spanish)
Figure 11.Q17: English Mail
Discussion:
Respondents noted difficulty noticing the instruction box directing the reader to Question 19 when
checking the “no” box.
Recommendation:
Consider moving the instruction box closer to the “No” and changing wording to be more deliberate such
as: “If you answered no, proceed to Question 19”.
45
Question 36— Mentioned 2 times (1x English, 1x Spanish)
Figure 12.Q36: English Mail
Discussion:
Respondents noted they understood the question to mean they should select a single option from the subquestion, as opposed to answering yes/no for each single option.
Recommendation:
Consider an instruction box that instructs respondent to check Yes and No for each sub-question if it
applies to them, similar to the instructions included in Q3 and Q53.
46
Question 41— Mentioned 2 times (1x English, 1x Spanish)
Figure 13.Q41: English Mail
Discussion:
One respondent noted there were two questions that Q41 lacked answer choices that applied to her. She
noted that she and her spouse have not yet made a decision about birth control given that they are only in
the early stages of the postpartum period. The wording of the question implies that some mothers are
answering “no” out of a personal choice.
One respondent thought the arrow pointing toward the instruction box instructing her to go to Q43 was
meant for her answer of No.
Recommendation:
Consider rephrasing the question to specifically ask if a method of birth control is being used, rather than
asking if they are doing anything to prevent pregnancy.
Consider creating more space and greater distance between Yes / No boxes and arrows, so that it is
visually more bold and clear which direction to go in for each answer. Also, include additional instruction
within instruction box such as: “If you answered Yes, go to Question X”.
47
Question 44— Mentioned 2 times (2x English)
Figure 14. Q44: English Mail
Discussion:
Respondents noted that they had not attended this appointment since they recently gave birth, however
they have appointments scheduled. They did not know to include this information, as it seems that they
had missed this appointment although they are just not far enough in their postpartum period. However,
one participant also missed the instruction box for answering no instructing her to proceed to Q46.
Recommendation:
Consider moving the instruction box closer to the No and changing wording to if answered No, skip
Question 45, Go to Question 46.
48
Question 5— Mentioned 2 times (2x Spanish)
Discussion:
See discussion for Question 4.
Recommendation:
See recommendation for Question 4.
49
Question 6— Mentioned 2 times (2x Spanish)
Figure 15.Q6: English Mail
Discussion:
One respondent felt Q6, Q7, and Q8 were repetitive. These three questions are about the exact same topic
(health insurance) but are asking about 3 different periods of time (before, during, and after pregnancy).
One respondent explained that she could not find an option that applied to her situation. She did not have
private health insurance, Medicaid, or another type of health insurance but she did have health coverage
through the hospital where she was receiving medical care. She described this as a plan that she had to
submit an application to and provide proof of income to qualify and receive discounted rates.
Recommendation:
Consider moving each question to the appropriate section of the survey since the survey is defined by
time periods.
Consider adding an option that includes this type of health benefit.
50
Question 7— Mentioned 2 times (2x Spanish)
Figure 16.Q7: Spanish Mail
Discussion:
One respondent felt Q6, Q7, and Q8 were repetitive. These three questions are about the exact same topic
(health insurance) but are asking about 3 different periods of time (before, during, and after pregnancy).
One respondent explained that she could not find an option that applied to her situation. She did not have
private health insurance, Medicaid, or another type of health insurance but she did have health coverage
through the hospital where she was receiving medical care. She described this as a plan that she had to
submit an application to and provide proof of income to qualify and receive discounted rates.
Recommendation:
Consider moving each question to the appropriate section of the survey since the survey is defined by
time periods.
Consider adding an option that includes this type of health benefit.
51
Question 8— Mentioned 2 times (2x Spanish)
Figure 17.Q8: Spanish Mail
Discussion:
One respondent felt Q6, Q7, and Q8 were repetitive. These three questions are about the exact same topic
(health insurance) but are asking about 3 different periods of time (before, during, and after pregnancy).
One respondent explained that she could not find an option that applied to her situation. She did not have
private health insurance, Medicaid, or another type of health insurance but she did have health coverage
through the hospital where she was receiving medical care. She described this as a plan that she had to
submit an application to and provide proof of income to qualify and receive discounted rates.
There is also a translation error, with “tenía” being the past tense of have, when this question is asking
about the current time.
Recommendation:
Consider moving each question to the appropriate section of the survey since the survey is defined by
time periods.
Consider adding an option that includes this type of health benefit.
Change “tenía” to “tiene” in this question.
52
Core Testing Question Abbreviated Analysis
As previously discussed, 18 of 30 questions were mentioned two or more times by different participants.
In this section, the remaining 12 of 30 questions will be discussed in an abbreviated format. The full
interviewer notes from the participants can be found in Appendix A- Core Testing Final Summary Report.
Question(s)
Discussion
Q19, Q33,
Participant experienced difficulty noticing or
Q42, Q43,
understanding the instruction boxes after answering
Q55
yes/no
Q37, Q38
Participant were confused by the term “new baby”.
Whether this is the first 2 weeks after birth or the
most recent two weeks.
Q56, Q57
Participant did not understand the term “ingreso” in
reference to income.
Q13
Participants received 2 COVID vaccinations prior to
pregnancy and 1 booster during pregnancy. She did
not know how to answer this question.
Q18
Participant did not understand the meaning of
“senales de advertencia”, and did not realize this
question was related to Q17.
Q32
Participant she stated she was unsure if the question
was asking whether her baby stayed in the hospital
extra days without her (she thought about babies that
go to NICU) or if the question referred to the time
period they were both in the hospital together.
53
Recommendation
Bring instruction box closer to
the yes/no options and possibly
make bolder to make
instructions more noticeable.
Consider making changes to the
wording of this section in
general. Remove the term new
baby and replace it with baby or
infant. And replace new baby
with newborn for questions that
are pertaining to the time period
in the hospital after baby was
born or right after coming home
from hospital.
Change “ingreso” to either
“salario” or “sueldo” which are
more common terms in Spanish
across different countries.
Consider using clearer language
or include instructions that you
are able to select more than one
box (before, during, after), for
each option.
Add the same definition for
warning signs that is included
in Q17. Or make Q17 and 2part question.
Consider rewording question to
make clear that the question
means how long did a mother
and her baby stay together at
the hospital post-birth.
7.0
CONCLUSIONS
The results presented in this report largely attest to the appropriateness of the maternal
and child health questionnaire (core). Appropriateness was boldly demonstrated in
respondents’ unanimous acknowledgement of ease in choosing answers. In addition, an
overwhelming majority of respondents reported that they had no difficulty in
comprehending the questions. Furthermore, spoken language of respondents, education
and race are independent of respondents’ opinions regarding quality of instructions,
difficulty/ease in the comprehension of questions and difficulty/ease in making choices
of answers. However, it is rather noticeable that perceived quality of instructions was not
robust. This suggests that there is a considerable room for improvement, as also
supported by the qualitative data.
8.0 RECOMMENDATIONS
The following are recommendations from the data:
1. Consider changing the position of instruction boxes between questions 4 and 5;
27, 28 and 29; 42 and 48 to the top of the next column or within question box.
2. Consider moving questions 6, 7 and 8 to the appropriate section of the
questionnaire since the questionnaire is defined by time periods (before, during
and after pregnancy).
3. In question 8, consider re-translating the word “tenia” being the past tense of have.
This question is asking about current so the correct word is “tiene”. The corrected
statement should read: ¿Qué tipo de seguro médico tiene usted ahora?
4. In question 11, consider adding answer choice option do not remember or do not
recall.
5. In question 32, consider rewording to make clear that the question means how
long did a mother and her baby stay together at the hospital post-birth.
6. In question 53, consider providing a brief explanation of postpartum care.
7. For questions 56 and 57, consider adding and/or changing ingreso total to salario
or sueldo, which are more commonly used terms to define income made.
54
8. For questions 36, 39 and 40, add an instruction box that instructs respondent to
check Yes and No for each sub-question if it applies to them, similar to instruction
included in questions 3 and 53.
9. For question 53, move the instruction outside of the main question to its own
instruction box to make more clear an answer is needed for each sub-question.
10. For the instruction box prior to question 16, add and/or to the instruction
statement to clarify confusion between before or during pregnancy - If you had
high blood pressure before and/or during your pregnancy, go to Question Core
16, if not, go to Question Core 17
11. The introduction to the section and questions 44, 46, 47, 48, 49, 51, 56, and 57
includes the statement: since your new baby was born. This statement implies a
baby was born and remains alive. A statement should be added to indicate
sensitivity to the fact that the child might not be alive.
12. In question 41, consider creating more space and greater distance between Yes/No
boxes and arrows, so that it is visually bolder and clearer which direction to go for
each answer.
13. To get rid of survey fatigue, consider reducing the number of questions.
55
Table of Appendixes
Appendix 1: Case summaries of all respondents (N = 10) .......................................................................... 57
Appendix 2: The Code of Ambiguity of Instructions ................................................................................... 58
Appendix 3: The Code of Clarity of Instructions ......................................................................................... 61
Appendix 4: The Code of Difficulty in Understanding Questions ............................................................... 63
Appendix 5: The Code of Ease in Understanding Questions ...................................................................... 65
Appendix 6: The Code of Difficulty in Choosing Answers ........................................................................... 67
Appendix 7: The Code of Ease in Choosing Answers .................................................................................. 69
Appendix 8: The Code of Recommendations Regarding Ambiguous Instructions ..................................... 71
Appendix 9: The Code of Recommendations Regarding Difficulty in Understanding Questions ............... 74
Appendix 10: The Code of Recommendations Regarding Difficulty in Choosing Answers ........................ 76
Appendix 11: The Code of Comments or Suggestions for Improving the Survey....................................... 77
56
Appendix 1: Case summaries of all respondents (N = 10)
Name
1
2
3
4
5
6
7
8
9
10
Total N
Erica
Respondent_I Age Race
D
BFA-MI-001
34 Hispanic
Languag Highest_Educat Household_Income
e
ion
English Greater than $85,001 and above
High School
Black Non- English Greater than $50,001 to $85,000
Hispanic
High School
Black Non- English High School
Less than $25,000
Hispanic
White Non- English Greater than $85,001 and above
Hispanic
High School
Black Non- English Greater than $25,001 to $50,000
Hispanic
High School
Hispanic
Spanish Less than High Less than $25,000
School
0
Spanish High School
$25,001 to $50,000
Hispanic
Spanish High School
$25,001 to $50,000
Hispanic
Spanish High School
$25,001 to $50,000
Hispanic
Spanish High School
0
9
10
10
9
Age_of_the_younges Date_of_Birth
t_child
4 months
03/17/1988
Bella
BFA-MI-002
0
10 months
11/09/1981
Bree
BFA-MI-003
22
1 month
01/29/2000
Meredith
BFA-MI-004
38
3 weeks
03/01/1984
Bailey
BFA-MI-005
31
2 weeks 1 day
03/12/1991
Catalina
BFA-MA-001
22
5 days
12/09/2000
Luca
Esmeralda
Juanita
Andrea
10
BFA-MA-002
BFA-MA-003
BFA-MA-004
BFA-MA-005
10
36
36
28
29
9
19 days
1 week
5 days
3 days
10
10/08/1985
07/26/1986
06/16/1994
01/10/1993
10
57
Appendix 2: The Code of Ambiguity of Instructions
Name: Nodes\\Ambiguity of Instructions
- § 1 reference
coded [23.29% Coverage]
Reference 1 - 23.29% Coverage
She did experience some difficulty with a few questions.
Respondent missed instruction box between Q27 and Q28. She noted she did not notice or read it.
For Q36, Q39, Q40, Q53, and Q55 respondent noted she understood the questions to mean she was
only to respond the one way in which the question applied to her. She wanted to just answer Yes for the
one way that she puts her baby to sleep in Q36, Q39 and Q40. And in Q53 she was overwhelmed by the
amount of options and felt compelled to just choose the one that she felt applied to her. For Q55, she
read the main questions and circled No because she has never experienced discrimination, and then saw
that none of the individual situations applied to her either so she left her originally marked answer.
Survey fatigue was also noted as a factor with Q36, Q39, Q40, Q53, and Q55, she was becoming tired of
the questions and of having to mark each individual box.
- § 1 reference coded [7.20% Coverage]
Reference 1 - 7.20% Coverage
however she did miss instruction box between Q27 and Q28, causing her to answer Q28 which did not
apply to her.
- § 1 reference coded [12.23% Coverage]
Reference 1 - 12.23% Coverage
However, respondent did show difficulty with the instruction box between Q27 and Q28. She missed
seeing this instruction box and rather than skipping Q28 to proceed to Q29, since she had not consumed
any alcohol during her pregnancy, she answered Q28. Similar experience occurred between Q42 and
Q43.
- § 1 reference coded [32.05% Coverage]
Reference 1 - 32.05% Coverage
58
Respondent experienced difficulty repeatedly with instruction boxes throughout survey. She missed
seeing instruction boxes between Q4 and Q5, as well as Q27 and Q28,
She also experienced difficulty with Q44, noting she has not yet had her postpartum checkup due to
only recently giving birth, however an appointment is scheduled for this checkup in the coming weeks.
In reading this question she wondered whether she should include this information and then missed the
instruction box with an arrow directing her to go to Q46 if she answered No. She noted it would have
been helpful to have this instruction box closer to the No, or worded differently. She also experienced
this difficulty with instruction box in Q17.
- § 1 reference coded [41.05% Coverage]
Reference 1 - 41.05% Coverage
Respondent experienced some difficulty with instruction boxes placed between Q4 and Q5. She noted
while she did notice and read the instruction box, she did not fully comprehend instruction and
therefore proceeded to the next question even though due to her answers to Q4 she should have
proceeded to Q6. She reported the instructions that included an arrow with instruction to go to a
specific question were more helpful and easier to understand than the instruction boxes.
Respondent also experienced difficulty with Q27, as she did not proceed to answer No for each subquestion, but rather circled No at the top of the column. She noted she simply answer No to the main
question because she did not drink alcohol at all throughout her pregnancy and skipped reading the rest
of the options in order to proceed to next question.
- § 1 reference coded [20.99% Coverage]
Reference 1 - 20.99% Coverage
with some mild confusion experienced with instructions that offered arrows next to Yes / No answer
choices. Respondent expressed that arrows pointing away from the No provide clear direction on where
to go next, because it points to an instruction box that states - Go to Question X. However, the arrow
pointing away from the Yes answer choice box does not have a similar instruction box, therefore she
was left to wonder whether she was to keep going and whether she was to keep answering all questions
that followed or only some questions. She assumed she was being directed to the very next question but
with a sense of doubt about her decision to keep moving forward in this manner.
Respondent also noted difficulty with instruction box on page 4, before Q16 -If you had high blood
pressure before or during your pregnancy, go to Question Core 16, if not, go to Question Core 17.
Respondent stated she had to re-read instruction several times to determine whether it was asking her
to proceed to Question Core 16 only if she had high blood pressure during one of the two time periods
(before or during pregnancy) or if she had high blood pressure during one of the two as well as both
time periods.
59
- § 1 reference coded [23.46% Coverage]
Reference 1 - 23.46% Coverage
She did miss the instruction box between Q15 and Q16, noting that she did not visually notice and
therefore did not read instructions. This occurred again with the instruction box between Q27 and Q28.
She also experienced difficulty with instructions on Q33, proceeding to Page 8 and not answering the
next set of questions on that page. She noted at this point survey fatigue was becoming significant and
she read the instruction box rather quickly, only noticing the beginning of the instruction.
For Q39 and Q40 wording of the questions and lack of instruction caused respondent to interpret that
she was only to answer Yes to the option that applied to her, and therefore she left all No check boxes
blank.
For Q41 her answer was No to this question, however she failed to mark it. She noted that while she did
not mark it, she did follow instructions but incorrectly. She thought the arrow pointing toward the
instruction box instructing her to go to Q43 was meant for her answer of No. Therefore, she skipped
Q42 and went to Q43, but left it blank because the question did not apply to her. She also noted survey
fatigue was a factor here.
- §
1 reference coded [26.53% Coverage]
Reference 1 - 26.53% Coverage
Respondent experienced difficulty throughout the survey with instruction boxes between questions, as
well as Yes / No instructions with arrows pointing out to instruction boxes. Respondent exhibited mixed
experience, fully understanding instructions for some questions and other times experiencing confusion.
Difficulty occurred with instruction boxes between Q4 and Q5, Q15 and Q16, Q19 Y/N instruction box
with arrow, Q23 Y/N instruction box with arrow.
- § 1 reference coded [8.82% Coverage]
Reference 1 - 8.82% Coverage
Respondent noted she did not notice the section headings Before / During / After Pregnancy. She
suggested making the whole heading of the page a defined section rather than the column alone as a
way to make this more visually clear for respondents.
60
Appendix 3: The Code of Clarity of Instructions
Name: Nodes\\Clarity of Instructions
- §
1 reference coded [2.48% Coverage]
Reference 1 - 2.48% Coverage
Respondent noted in general she was able to clearly understand instructions throughout the survey.
- § 1 reference coded [15.35% Coverage]
Reference 1 - 15.35% Coverage
Respondent noted survey was “well organized and noticed the distinction of the marked sections Before
/ During / After Pregnancy, which helped her remain in that period of time. She expressed she generally
noticed all of the instruction boxes.
- § 1 reference coded [6.99% Coverage]
Reference 1 - 6.99% Coverage
Respondent noted instructions were generally clear and easy to understand. Instruction boxes were
generally noticeable and “straightforward”, assisted with flow of survey.
- § 1 reference coded [1.04% Coverage]
Reference 1 - 1.04% Coverage
Respondent stated instructions were generally understandable.
- § 2 references coded [32.05% Coverage]
Reference 1 - 12.67% Coverage
Respondent stated she was able to understand instructions clearly throughout the survey and noticed
the section headings which allowed her to move from question to question easily.
61
Reference 2 - 19.38% Coverage
She did note that personally she likes to read all questions thoroughly to make sure that she is
answering questionnaires correctly, so this affected her with mistakenly answering Questions 24, 25,
and 26 even though her answer to Q23 meant she should have skipped to Q27.
- § 1 reference coded [3.10% Coverage]
Reference 1 - 3.10% Coverage
Respondent noted instructions were generally clear for her to understand and follow, instructions with
arrows in particular were identified as helpful.
- § 1 reference coded [5.24% Coverage]
Reference 1 - 5.24% Coverage
Respondent noted instructions were clear to follow. She reported in general the instructions helped her
to understand where to go next in the survey.
62
Appendix 4: The Code of Difficulty in Understanding Questions
Name: Nodes\\Difficulty in understanding questions
- §
1 reference coded [14.31% Coverage]
Reference 1 - 14.31% Coverage
She did experience some confusion with a couple questions.
For Q18 she was not sure what was meant by “las señales de advertencia” expressing that she did not
connect Q18 to Q17 and therefore did not understand that the warning signs being asked about in Q18
were the same one which were defined in Q17.
For Q32, she stated she was unsure if the question was asking whether her baby stayed in the hospital
extra days without her (she thought about babies that go to NICU) or if the question referred to the time
period, they were both in the hospital together.
- § 1 reference coded [22.88% Coverage]
Reference 1 - 22.88% Coverage
Respondent did report experiencing some confusion with Questions 36-40 which asked about baby’s
sleeping in the past 2 weeks. She noted due to Q32 and instruction box - If your baby is still in the
hospital, go to Question Core 41, she wondered whether she was supposed to think about the first 2
weeks after bringing baby home from hospital or if she was supposed to think about the most recent 2
weeks. Note that respondent’s baby is now 10 months old. Respondent noted she decided to answer
the question thinking about the 2 weeks prior to the current date.
- § 1 reference coded [22.09% Coverage]
Reference 1 - 22.09% Coverage
However, she did report having difficulty with Q11, noting that the Ask me… section of the question
caused confusion about how to answer the question. The framing of the question in the first person Ask
me… if I was drinking alcohol caused respondent to question whether she was supposed to answer No
because she had not consumed alcohol during her pregnancy, or Yes because her doctor had spoken
with her about this topic. She experienced similar confusion for sub-questions g, h, i, k, and l.
63
- § 1 reference coded [12.77% Coverage]
Reference 1 - 12.77% Coverage
She stated she did have to re-read some questions a few times to ensure that she was thinking of the
correct time frame i.e., 12 months before you got pregnant, before or during your pregnancy, last 3
months of your pregnancy. Respondent also noted experiencing some confusion and difficulty with fully
understanding what was meant by postpartum care in Q53. She expressed that in her experience
postpartum care can mean a variety of things including care at the hospital, care by lactation
consultants, care from one’s OBGYN, as well as in her case since her baby was in NICU, the postpartum
care received by herself and her baby during that time. Therefore, she wondered which one of these the
question was asking her to think about.
- § 1 reference coded [16.03% Coverage]
Reference 1 - 16.03% Coverage
However, she did report becoming confused with Q3-a and stated she had to re-read several times to
ensure she was interpreting correctly whether the question was asking about type 1 or type 2 diabetes,
or gestational diabetes.
- § 1 reference coded [12.98% Coverage]
Reference 1 - 12.98% Coverage
Although there were a few that were confusing and a few that were repetitive so she thought the same
question was being asked which caused her to leave blank answers. For Q56, she did not understand the
meaning of “ingreso total”. For Q57, while she understood the question fully, she did not understand
what was being referred to when it stated “dependian de este dinero” because Q56 and Q57 were not
connected in her mind.
Q6, Q7, and Q8 were identified as repetitive.
Additionally, she expressed the survey had too many questions, and she began to experience a desire to
finish as quickly as possible midway through the survey.
- § 1 reference coded [7.57% Coverage]
Reference 1 - 7.57% Coverage
She did note Q13 caused some minor confusion related to the Covid19 vaccination being that she
received two doses prior to pregnancy and a booster shot during pregnancy so she was unsure how to
answer this question.
64
Appendix 5: The Code of Ease in Understanding Questions
Name: Nodes\\Ease in understanding questions
- §
1 reference coded [2.00% Coverage]
Reference 1 - 2.00% Coverage
Respondent noted overall she found questions to be easy and clear to understand.
- § 1 reference coded [4.61% Coverage]
Reference 1 - 4.61% Coverage
Respondent noted overall she found questions to be easy to understand.
- § 1 reference coded [2.74% Coverage]
Reference 1 - 2.74% Coverage
Respondent noted in general question were easy to understand.
- § 1 reference coded [3.19%
Coverage]
Reference 1 - 3.19% Coverage
Respondent noted questions were generally easy to understand.
- § 1 reference coded [1.06% Coverage]
Reference 1 - 1.06% Coverage
Respondent noted questions were generally easy to understand.
- § 1 reference coded [4.34% Coverage]
Reference 1 - 4.34% Coverage
Respondent noted in general question were easy to understand.
65
- § 1 reference coded [1.05% Coverage]
Reference 1 - 1.05% Coverage
Respondent stated she generally understood questions.
- § 1 reference coded [3.73% Coverage]
Reference 1 - 3.73% Coverage
Respondent stated questions were clear and easy to understand.
- § 1 reference coded [2.08% Coverage]
Reference 1 - 2.08% Coverage
Respondent noted questions were “clear and straightforward”.
66
Appendix 6: The Code of Difficulty in Choosing Answers
Name: Nodes\\Difficulty in choosing answer
- §
1 reference coded [12.16% Coverage]
Reference 1 - 12.16% Coverage
She did state not having an answer choice that applied to her situation for Q6, Q7, and Q8, therefore
leaving these blank.
Respondent explained that she did not have private health insurance, Medicaid, or another type of
health insurance but she did have health coverage through the hospital where she was receiving medical
care. She described this as a plan that she had to submit an application to and provide proof of income
in order to qualify and receive discounted rates.
- § 1 reference coded [4.71% Coverage]
Reference 1 - 4.71% Coverage
However, she did state that the questions that offered Likert scale answer options were mildly difficult
for her noting that she finds the words Sometimes and Rarely to be almost the same. She suggested
removing the rarely or replacing with something like hardly at all.
- § 1 reference coded [3.93% Coverage]
Reference 1 - 3.93% Coverage
However, she did have some difficulty with some answer choice options.
For Q11 she left several blank as she was unsure/could not remember if these topics were brought up by
her provider.
- § 1 reference coded [23.90% Coverage]
Reference 1 - 23.90% Coverage
She noted there were two questions that were lacking answer choices that applied to her situation,
given that her baby is a 3-wk old newborn. Q41 and Q44 were identified. Respondent noted that she
and her spouse have not yet made a decision about birth control given that they are only in the early
stages of the postpartum period. So, she has not yet made a decision about birth control not because
she does not want to but because she just had her baby. Similarly, she has not attended a postpartum
checkup yet, but has one scheduled. She shared that answering No to Q44 felt as if she had perhaps
67
“missed” this appointment and not because she was too early in her postpartum period.
68
Appendix 7: The Code of Ease in Choosing Answers
Name: Nodes\\Ease in choosing answers
- §
1 reference coded [1.65% Coverage]
Reference 1 - 1.65% Coverage
Respondent noted in general having no trouble with answer choices.
- § 1 reference coded [21.60% Coverage]
Reference 1 - 21.60% Coverage
Respondent noted she appreciated the detailed and long list of option choices provided. She also
noticed and appreciated the attention to inclusive frame and language used throughout the survey. For
example, the question related to birth control method in which “We are same-sex spouses/partners”
was provided as an answer choice option.
- § 1 reference coded [2.93% Coverage]
Reference 1 - 2.93% Coverage
Respondent noted having no trouble choosing between answer choices.
- § 1 reference coded [5.22% Coverage]
Reference 1 - 5.22% Coverage
Respondent noted answer choices were generally easy to understand and had no trouble choosing
between answer choices.
- § 1 reference coded [4.07%
Coverage]
Reference 1 - 4.07% Coverage
Respondent stated she generally had no trouble choosing between options given.
- § 1 reference coded [1.35% Coverage]
69
Reference 1 - 1.35% Coverage
Respondent stated she generally had no trouble choosing between options given.
- § 1 reference coded [8.75% Coverage]
Reference 1 - 8.75% Coverage
Respondent noted answer choices were generally easy to understand and had no trouble choosing
between different options.
- § 1 reference coded [2.30% Coverage]
Reference 1 - 2.30% Coverage
Respondent expressed in general answer choices were easy to pick from and did not experience
significant trouble.
- § 1 reference coded [4.40% Coverage]
Reference 1 - 4.40% Coverage
Respondent did not experience difficulty choosing between answer choices.
- § 1 reference coded [3.82% Coverage]
Reference 1 - 3.82% Coverage
Respondent expressed in general answer choices were easy to choose from and related back well to the
question.
70
Appendix 8: The Code of Recommendations Regarding Ambiguous Instructions
Name: Nodes\\Ambiguity of Instructions\Recommendations regarding ambiguous instructions
- § 1 reference
coded [23.51% Coverage]
Reference 1 - 23.51% Coverage
Recommendations
Change position of instruction box between Q27 and Q28 to top of column or within question box. For
instruction boxes between questions, consider adding a more bold definition to the instruction box so
that it is more visually noticeable. Add bold letters, shadows, colors or add Please Read notices within
instruction boxes to make more clear an instruction will change the flow of answering questions.
For Q36, Q39 and Q40: add an instruction box that instructs respondent to check Yes and No for each
sub-question if it applies to them, similar to instruction included in Q3 and Q53 where it states “Para
cada una marque No, si no….. o Si, si …….”
For Q53 move the instruction outside of the main question to its own instruction box to make more
clear an answer is needed for each sub-question.
For survey fatigue: consider reducing number of questions or breaking up the survey into two parts.
- § 1 reference coded [6.00% Coverage]
Reference 1 - 6.00% Coverage
Change position of instruction box between Q27 and Q28 to top of column or within question box.
- § 1 reference coded [4.27% Coverage]
Reference 1 - 4.27% Coverage
Change location of instruction box between Q28 and Q29, and Q42 and Q43 to the top of the next
column.
- § 1 reference coded [11.73% Coverage]
Reference 1 - 11.73% Coverage
Change location of instruction box between Q4 and Q5, and Q27 and Q28 to the top of the next column
or to the top of the question.
Consider moving the instruction box closer to the No and changing wording to if answered No, skip
Question 45, Go to Question 46.
71
- § 1 reference coded [24.96% Coverage]
Reference 1 - 24.96% Coverage
Consider changing position of the instruction box to the top of Q4 so that it is visible prior to answering
the question.
Or, change instructions to provide more guidance about how to proceed, for example: “If you answered
No to each question and did not attend any medical appointments in the 12 months prior to becoming
pregnant, proceed to Question 6. “Si usted no tuvo una cita de atención médica en los 12 meses antes de
su embarazo y contestó NO a todas las preguntas pase a la pregunta número 6.”
- § 2 references coded [27.68% Coverage]
Reference 1 - 14.06% Coverage
For questions with Yes / No answer choices For Yes / No answer choice options with arrows pointing to
instruction boxes away from the answer choices, consider adding an instruction box for the Yes answer
choice as well so that respondents are clear about proceeding to the very next question.
For the instruction box prior to Q16, add and/or to the instruction statement to clarify confusion
between before or during pregnancy - If you had high blood pressure before and/or during your
pregnancy, go to Question Core 16, if not, go to Question Core 17
Or
Rephrase statement entirely, for example:
If you were diagnosed with high blood pressure before becoming pregnant or you had high blood
pressure during your most recent pregnancy, go to Question Core 16, if not go to Question Core 17.
Reference 2 - 13.62% Coverage
Additionally, respondent’s remarks should be taken into consideration for changes to be made in
general to the After Pregnancy section.
The introduction to the section and Questions 44, 46, 47, 48, 49, 50, 51, 56, and 57 all include that
statement: since your new baby was born. This statement implies a baby was born and remains alive,
therefore mothers who have experienced the death of their babies and are completing the survey would
be placed in a position where a grief response could be triggered in a manner that is difficult to manage
and she would be provided no immediate support to comfort that response.
The same is true for questions prior to this section that include the statements: before you got pregnant
with your new baby and before your new baby was born.
- § 1 reference coded [17.34% Coverage]
Reference 1 - 17.34% Coverage
72
For instruction boxes between questions, consider adding a bolder definition to the instruction box so
that it is more visually noticeable. Add bold letters, shadows, colors or add Please Read notices within
instruction boxes to make clearer an instruction will change the flow of answering questions.
For Q39 and Q40: add an instruction that instructs respondent to check Yes and No for each subquestion if it applies to them, similar to instruction included in Q3 and Q53 where it states “Para cada
una marque No, si no….. o Si, si …….”
For Q41: consider creating more space and greater distance between Yes / No boxes and arrows, so that
it is visually more bold and clear which direction to go in for each answer. Also, include additional
instruction within instruction box such as, If you answered Yes, go to Question X.
- §
1 reference coded [36.23% Coverage]
Reference 1 - 36.23% Coverage
Change the location of instruction boxes between questions to top of column or to area below question
and before answer choices to make clearer that a question will be skipped when questions are answered
in a certain manner.
Create definition between questions, answer choices and instruction boxes with bold letters, shadows,
colors or add Please Read notices within instruction boxes to make clearer an instruction will change the
flow of answering questions.
Include instruction boxes with arrows for both Yes and No, and add sentences with instructions such as
if answered Yes, go to Question X. If answered No, go to Question Y.
73
Appendix 9: The Code of Recommendations Regarding Difficulty in Understanding
Questions
Name: Nodes\\Difficulty in understanding questions\Recommendation regarding difficulty in
understanding questions
- §
1 reference coded [6.68% Coverage]
Reference 1 - 6.68% Coverage
For Q18: add the same definition for warning signs that is included in Q17. Or make Q17 and 2-part
question.
For Q32: consider rewording question to make clear that the question means how long did a mother and
her baby stay together at the hospital post-birth.
- § 1 reference coded [20.93% Coverage]
Reference 1 - 20.93% Coverage
Consider making changes to the wording of this section in general. Remove the term new baby, and
replace it with baby or infant. And replace new baby with newborn for questions that are pertaining to
the time period in the hospital after baby was born or right after coming home from hospital.
** in Spanish the terms would be: newborn is recien nacido, infant is bebe.
Respondent made suggestion to change phrasing of questions to: In the most recent two weeks, at your
child’s current age, ______.
- § 1 reference coded [2.92% Coverage]
Reference 1 - 2.92% Coverage
Consider changing sentence structure to:
Ask you… if you
- § 1 reference coded [2.03% Coverage]
Reference 1 - 2.03% Coverage
Define postpartum care to include time period and type of postpartum care that is being referred to in
this question.
74
- § 1 reference coded [20.12% Coverage]
Reference 1 - 20.12% Coverage
For Q56 and Q57: add and/or change ingreso total to salario or sueldo, which are more commonly used
terms to define income made. This is especially true for mothers that may not complete taxes, as
“ingreso total” will not be a term that is easily recognized outside of tax preparation. Also, add the
definition of what you mean by “ingreso total”, such as “the total amount of money you and/or your
spouse/partner earned from your job”.
For Q6, Q7, and Q8: note that these three questions are about the exact same topic (health insurance)
but are asking about 3 different periods of time - before, during and after pregnancy. Consider moving
each question to the appropriate section of the survey since the survey is defined by time periods.
Additionally Q8 has a translation error with word “tenia” being the past tense of have, when this
question is asking about current so the correct word is “tiene”. The corrected statement should read:
¿Qué tipo de seguro médico tiene usted ahora?
75
Appendix 10: The Code of Recommendations Regarding Difficulty in Choosing
Answers
Name: Nodes\\Difficulty in choosing answer\Recommendation regarding difficulty in choosing
answers
- §
1 reference coded [2.08% Coverage]
Reference 1 - 2.08% Coverage
For Q6-8: consider adding an option that includes this type of health benefit.
- § 1 reference coded [1.69% Coverage]
Reference 1 - 1.69% Coverage
For Q11: consider adding answer choice option Do not remember or Do not recall.
- § 1 reference coded [24.31% Coverage]
Reference 1 - 24.31% Coverage
The questions are presented in a manner that implies to some mothers they are answering No out of
personal choice, which will perhaps be viewed in a biased manner by someone reading responses,
rather than answering No because it is not time yet to consider these decisions. Consider reframing the
wording of the questions and/or add answer choice options that specifically reflect the experiences of
mothers in that very early stage (1day to 4 weeks) of the postpartum period.
Examples: rather than stating “Are you or your spouse doing anything now to keep from getting
pregnant”, which implies choice and judgment use “ Are you and/or your spouse currently using a
method of birth control?”
76
Appendix 11: The Code of Comments or Suggestions for Improving the Survey
Name: Nodes\\Comments or suggestions for improving the survey
- §
1 reference coded [1.43% Coverage]
Reference 1 - 1.43% Coverage
Respondent made no additional comments or suggestions.
- § 1 reference coded [14.09% Coverage]
Reference 1 - 14.09% Coverage
Respondent noted possibility of adding more questions related to how people become pregnant to
include experiences of same-gender loving couples as well as others who experience various paths
towards becoming pregnant.
- § 1 reference coded [7.27% Coverage]
Reference 1 - 7.27% Coverage
Respondent suggested survey should be made available in an electronic/online format as well as paper
format to give mothers options to choose from as well as for convenience.
- § 1 reference coded [2.79% Coverage]
Reference 1 - 2.79% Coverage
Respondent did not share additional comments or suggestions.
- § 1 reference coded [2.75%
Coverage]
Reference 1 - 2.75% Coverage
Respondent made no further comments or suggestions.
- § 1 reference coded [20.26% Coverage]
77
Reference 1 - 20.26% Coverage
Respondent shared about her own past experience of the death of a child and suggested considering the
emotional impact that answering questions about pregnancy after a baby has died will have on a
grieving mother.
In particular she noted after Q33, a change should be made in how mothers taking the survey are
instructed to continue. She noted that while the statement We are very sorry for your loss is noticeable,
there is a lack of empathy for how a mother who has experienced the death of a baby will be impacted
from this point forward with the survey. She noted some mothers may not want to continue after this
question. And for mothers that do continue, even though the instruction box provides the direction to
go to page 7, Q41 a mother may still skim through the questions that ask about the baby’s sleep while
trying to find Q41, and those questions may be difficult to read after one has been asked to check Yes
for Q33.
Respondent suggested creating a blank space after Q33 and keeping Questions 34-40 within its own
distinct section, so that when a mother turns to the page where she is to resume the survey the first
question on that page is Q41.
- § 1 reference coded [5.60% Coverage]
Reference 1 - 5.60% Coverage
Respondent did not make additional comments or suggestions about the survey.
- § 1 reference coded [4.56% Coverage]
Reference 1 - 4.56% Coverage
Respondent suggested making survey length shorter, she noted taking into consideration mother’s sleep
deprivation and taking care of children as factors to not making surveys so long and repetitive with some
of the questions.
- § 1 reference coded [3.62% Coverage]
Reference 1 - 3.62% Coverage
Respondent did not offer additional suggestions or comments.
- § 1 reference coded [7.68% Coverage]
Reference 1 - 7.68% Coverage
Respondent noted having an electronic/online version of survey would be helpful, especially for
questions that have if Yes go to Q__ if No go to Q__ because an electronic version would make this
78
choice for respondents.
79
| File Type | application/pdf |
| File Title | Microsoft Word - resized - Core An analytical report of field experiences - Formatted resized.docx |
| File Modified | 2023-02-14 |
| File Created | 2022-08-08 |