2023–24 NATIONAL POSTSECONDARY STUDENT AID STUDY (NPSAS:24)
FULL-SCALE STUDY
Student Data Collection and Student Records
Appendix H
Spanish Cognitive Testing Findings
OMB # 1850-0666 v. 36
Submitted by
National Center for Education Statistics
U.S. Department of Education
October 2023
Contents
Tables
Table 1. Participant Demographics H-4
Table 2. Name of institution attended during NPSAS academic year H-6
Table 3. N24BREMEVER – Taken any remedial courses since high school H-7
Table 4. N24ADIPL – Type of high school completion H-8
Table 5. N24ADEGREE - Degree program at NPSAS institution H-9
Table 6. N24ANENRLFT - Months of full-time attendance at NPSAS institution H-10
Table 7. N24EPREPAR - Prepare for teaching career at the preK - 12th grade level H-11
Table 8. N24EHOWPREP - Steps to prepare for teaching career at the preK - 12th grade level H-12
Table 9. N24BFUTWAGES - Future expected yearly wages H-13
Table 10. N24CAIDGATE1/ N24CAIDGATE2 - Financial aid gate for the NPSAS academic year H-14
Table 11. N24DGIGINTRO - Introduction to nonstandard and gig job section H-15
Table 12. N24DGIGLIST - Paid for any nonstandard or gig job H-16
Table 13. N24HINCOM - Income for prior calendar year from all sources H-17
Table 14. N24EFIN2000 - Confidence could come up with $2,000 within next month H-18
Table 15. N24FSX – Sex assigned at birth H-18
Table 16. N24FGENDERID - Gender identity H-19
Table 17. N24FLGBTQ - Sexual orientation H-19
Table 18. N24BEQUITABLE - Emphasis on equitable treatment at NPSAS institution H-20
Table 19. N24BVALUED - Value and belonging at NPSAS institution H-23
Table 20. N24BRACECOMP - Racial composition of pre-NPSAS community H-24
Table 21. N24HPARED1 - Parent 1: highest level of education completed H-25
Table 22. N24HPARST - Parents’/guardians’ marital status H-26
This report presents findings from cognitive testing of selected survey questions from the 2023-24 National Postsecondary Student Aid Study (NPSAS:24) Field Test Student Survey. Based on findings from NPSAS:20, testing was requested to help improve the Spanish survey experience in the NPSAS:24 study, particularly for students attending institutions in Puerto Rico who are most likely to take the Spanish survey. Results from cognitive testing will help refine the Spanish survey questions, maximize the quality of data collected, and provide information on issues with important implications for the design of the Spanish language survey, such as the following:
The comprehension of certain terms in survey questions, including updated and added terminology;
The thought processes used to arrive at answers to survey questions;
Patterns of interpretation for each item;
Improved translations of existing NPSAS survey questions; and
Sources of burden and participant stress.
Testing focused on two subsets of survey questions:
NPSAS:20 survey items with the largest discrepancy in timing between English and Spanish self-administration; and
Survey questions that are either new to NPSAS (i.e., gig economy) or revised (i.e., COVID-19 topics).
Specifically, this includes questions intended to collect data on participants’ months enrolled in postsecondary education and intensity; education experiences; financial aid receipt; employment experiences; and sexual orientation and gender identity.
The following Methods section covers the recruiting strategy, characteristics of the sample of participants, a detailed discussion of the findings, and study limitations.
Recruiting for 20 participants began at the end of December 2022. Several recruiting strategies and avenues were utilized, and some proved more successful than others:
Online contact, e.g., posting Facebook pages, e-mailing appropriate staff at universities in Puerto Rico, community colleges mainly in the Midwest, vocational education centers, libraries, and student organizations in areas with large Hispanic or Latino populations; and
In-person recruiting, e.g., posting printed flyers in community colleges and word-of-mouth efforts.
Contacting universities proved least successful, as university officials would often not respond to e-mails or calls. Similarly, individuals who said they were going to help identify recruits would often not follow up or refer us to individuals that could not help. The most successful strategy was through word-of-mouth, including students sharing information with their peers, followed by the assistance of community organizations and libraries.
To qualify for participation, each participant had to be a Spanish-speaking monolingual that attended a university, college, or trade school at any time since July 1, 2021.
The following question was used to determine eligibility based on language skills:
The interview will be completely in Spanish, and it is not necessary to speak English well. But, to classify participants, I need to ask you how well you speak English. Would you say you speak English very well, well, not very well, or not at all?
[“La entrevista será completamente en español y no es necesario saber hablar inglés. Pero para clasificar a los participantes, necesito preguntarte qué tan bien hablas inglés. ¿Dirías que hablas inglés muy bien, bien, no muy bien, o para nada?”]
To qualify, participants had to answer not very well or not at all.
As recruiting began, it was soon apparent that finding eligible participants would be challenging. Half of the callers during the first phase of recruitment had not attended school since July 1, 2021. In addition, many potential participants screened spoke English well or very well; many individuals living in Puerto Rico mentioned that they had an educational requirement to speak English.
In consultation with RTI, the recruitment approach was changed to try to capture a wider range of Spanish-speaking, or predominantly Spanish-speaking, individuals who had attended school any time since July 1, 2019. A revised screener was submitted to OMB with a change memo. Replacing the original screening question, the revised screener included two questions:
What language do you generally speak at home?
[“¿En qué idioma hablas generalmente en tu casa?”]
and
Think about completing a 30-minute survey about your experiences in college or university, that can be completed over the phone with an interviewer or by internet. Would you prefer to complete this survey in English or Spanish?
[“Piensa en completar una encuesta de 30 minutos sobre tus experiencias en el college o la universidad, que se puede completar por teléfono con un entrevistador o por internet. ¿Preferirías completar esa encuesta en inglés o en español?”]
Students qualified if: 1) they spoke only or mostly Spanish at home, or Spanish and English about equally, and 2) preferred to answer surveys in Spanish.
OMB approval for the change was received on March 31, 2023. Subsequently, recruiters recontacted individuals who had screened out in the first round of recruitment, but that would now be eligible. If still available, they were recruited into the study.
A total of 20 participants were interviewed in the present study. Table 1 provides a summary of participants’ demographic characteristics.
Table 1. Participant Demographics
Participants’ Demographics |
Percent
|
Gender |
|
Female |
0.80 |
Male |
0.20 |
Age |
|
18-24 |
0.45 |
25-29 |
0.10 |
30-34 |
0.15 |
35-39 |
† |
40-49 |
0.20 |
50-59 |
0.10 |
Race1 |
|
American Indian or Alaska Native |
0.10 |
Asian |
† |
Black or African American |
0.15 |
Native Hawaiian or Pacific Islander |
0.05 |
White |
0.60 |
Other race |
0.15 |
Hispanic, Latino, or Spanish Origin |
1.00 |
College, university, or vocational/technical school |
|
University (Puerto Rico) |
0.60 |
Community college (Midwest) |
0.05 |
Private 4-year college (Midwest) |
0.25 |
Technical School (Midwest, West Coast) |
0.10 |
Degree Program or Course Type2 |
|
Associate’s degree |
0.20 |
Bachelor’s degree |
0.45 |
Master’s degree |
0.15 |
Doctoral degree – professional practice |
† |
Doctoral degree – research/scholarship |
† |
Undergraduate certificate or diploma |
0.10 |
Graduate level classes |
† |
Undergraduate level classes |
0.10 |
Annual Income |
|
Less than $20,000 |
0.35 |
$20,000 to $49,000 |
0.45 |
$50,000 to $99,000 |
0.20 |
$100,000 or more |
† |
Prefer not to answer |
† |
† Not applicable. 1 Participants were able to report more than one race, therefore percents do not sum to 100. 2 These responses differ from the figures reported in the screening interview because three participants responded differently in the cognitive interview. The table above reflects the responses from the cognitive interview.
|
|
Although only one round of interviews was originally planned, after completing the first ten interviews, RSS communicated to RTI initial findings that indicated some items were not working as intended due to terminology issues. RSS and RTI agreed on the benefits of making non-substantive modifications to the Spanish questions and testing the revised wording in the second half of data collection. NCES agreed that these non-substantive tweaks were necessary to improve comprehension.
In this section, we present findings by question, for the 21-item questionnaire tested. For questions where a revised version was tested starting in April, both original (Version 1) and revised (Version 2) are listed, with the differences in question text highlighted.
Table 2. Name of institution attended during NPSAS academic year
Version 1 |
Version 2 |
¿Cuál es el nombre del último college, universidad o escuela vocacional o técnica donde has asistido desde el primero de julio de 2019?
(Si has asistido a más de una escuela ¿Cuál es el nombre de la escuela que consideras que es tu escuela principal?) |
¿Cuál es el nombre del último college, universidad o escuela vocacional o técnica donde has asistido después del primero de julio de 2019?
(Si has asistido a más de una escuela ¿Cuál es el nombre de la escuela que consideras que es tu escuela principal?) |
Note: This represents a combination of a NPSAS survey item (N24AELIG) and the collection of institution name for the purposes of populating a name fill for the cognitive interview questions. For the actual NPSAS:24 student survey, respondents will not be asked to provide the name of the institution as this is pre-populated during sampling. Therefore, the cognitive testing results are focused on the interpretation of the main wording stem, which N24AELIG will utilize.
Overall Question Interpretation
Twelve of the 20 participants attended a university in Puerto Rico, five attended a private 4-year college in the Midwest, two attended technical school, and one attended community college in the Midwest.
This question was posed in two versions. A slight change was made to Version 1 which said “desde” [since], to Version 2 saying “después” [after].
Most participants, regardless of which version they received, understood the question without difficulties. However, three participants had some trouble answering the question. Two participants, one in each version, were unclear that the dates in the question were intended to capture the period starting in July 2019 and up to the present. One participant who received Version 2 was confused because they attended two universities in that period, but selected the university they were currently attending.
Recommendations
We recommend using Version 2. Version 1 can make someone think the question is asking for the school they are currently attending, or the school they continued to attend since the date referenced.
Table 3. N24BREMEVER – Taken any remedial courses since high school
Version 1 |
Version 2 |
Los cursos remediales o clases preparatorias se usan para reforzar tus destrezas en matemáticas, lectura u otras materias. A los estudiantes usualmente se les asigna a estos cursos con base a una prueba de ubicación que se toma antes del inicio del año académico. Frecuentemente, los cursos remediales no cuentan como créditos para graduarse.
Desde que cumpliste con los requisitos de escuela superior ¿has tomado algún curso remedial o preparatorio para mejorar tus destrezas básicas en inglés, matemáticas, lectura, o redacción?
|
A veces a los estudiantes universitarios se les requiere que tomen cursos para reforzar sus destrezas en matemáticas, lectura u otras materias, antes de permitirles tomar cursos por crédito. Usualmente se les asigna a estos cursos con base a una prueba de colocación que se toma antes del inicio del año académico. A estos cursos se les llama cursos remediales o preparatorios. Frecuentemente, no cuentan como créditos para graduarse. Desde que cumpliste con los requisitos para graduarte de high school, ¿has tomado algún curso remedial o preparatorio para mejorar tus destrezas básicas en inglés, matemáticas, lectura, o redacción?
|
Overall Question Interpretation
Of the 20 participants, 10 answered No, 9 answered Yes, and 1 declined to answer.
Overall, participants did not appear to be familiar with the concept of “cursos remediales o clases preparatorias” [remedial or prep courses]. The definition provided in Version 1 did not appear to clarify the concept for several participants.
Some had heard of these types of classes by other terms, including “cursos de repaso”, and others had heard the term in the question but were not sure what “remedial” meant. One participant thought these were courses that students were placed in depending on their College Board scores, another participant thought these might be practice tests, while another thought these classes might be referring to the two years of school before beginning university. Another participant thought these might be skills developing classes, such as Excel courses.
Participants were probed on what a “prueba de ubicación” [placement test] was. This term was used differently in the two versions of the question. Version 1 used “prueba de ubicación,” while Version 2 used “prueba de colocación.” This change was based on findings in the first set of interviews that indicated some participants called placement tests “pruebas de colocación.”
While some participants understood the concept of placement tests, others had not heard of these. They thought these tests referred to required exams after high school, or classes to improve grades on a given course. Participants had also heard placement tests the names of “pruebas diagnosticas” and “pruebas de conocimiento.”
There were several changes made to the second version for clarity purposes. The information was presented in different order in the second version, offering more context to make it clear that remedial courses are not initiated by the student but by the institution. Version 2 used “pruebas de colocación” in lieu of “pruebas de ubicacion”, and the majority administered this version understood the term. Additionally, Version 2 used the term “high school” in lieu of “escuela superior,” which was more accessible to all participants, especially those outside of Puerto Rico. This also resolved an inconsistency in the translation, since in most other items the term appeared as “high school” (see further discussion below in ADIPL).
Recommendations
The second version was more clearly understood, so it is recommended that Version 2 be adopted. Additionally, the definition of placement test in question help text should mention alternative names that participants used: “pruebas diagnósticas” and “pruebas de conocimiento”.
Table 4. N24ADIPL – Type of high school completion
Version 1 |
Version 2 |
¿Cuál de las siguientes opciones describe mejor cómo terminaste la escuela superior?
o Recibí un diploma de escuela superior en los Estados Unidos o Recibí un certificado de GED (General Educational Development) u otra credencial equivalente (como HiSET o TASC) o Recibí un certificado de que completé la escuela superior o Asistí a una escuela superior en el extranjero o Asistí a escuela en el hogar o No terminé la escuela superior ni un programa de equivalencia de escuela superior |
¿Cuál de las siguientes opciones describe mejor cómo terminaste high school?
o Recibí un diploma de high school en los Estados Unidos o Recibí un certificado de GED (General Educational Development) u otra credencial equivalente (como HiSET o TASC) o Recibí un certificado de que completé high school o Asistí a high school en el extranjero o Asistí a escuela en el hogar o No terminé high school ni un programa de equivalencia de high school
|
Overall Question Interpretation
Ten of the 20 participants received a high school certificate, four received a high school diploma in the United States, four attended high school abroad, one participant received a GED or an equivalent credential, one participant was home schooled, and one did not answer because he thought no questions applied to him, although he had finished high school in Colombia.
Participants answered this question with no apparent difficulties. However, 18 of the 20 participants had never heard of “HiSET” and “TASC,” and some were unfamiliar with the GED. Two participants defined how they interpreted these terms: one thought these were exams administered by the state to earn your diploma, and another said that these were accelerated programs to earn your high school certificate in a short period of time. Among those who had heard of the GED, they were able to correctly interpret it as an equivalent to high school completion. One participant guessed that the “HiSET” and “TASC” were like the GED.
Participants were also asked to define “escuela en el hogar.” Most participants correctly interpreted it as homeschooling and volunteered the English term. However, three participants interpreted it as remote school or taking virtual classes.
One change was needed after testing Version 1, which replaced “escuela superior” with high school. The term “escuela superior” did not work well for participants of non-Puerto Rican origins.
While the term “escuela superior” worked well with participants from Puerto Rico and those that had been exposed to Puerto Rican students, other respondents had various interpretations, including university education. For participants of other national origins, this was not consistently interpreted as high school. While a participant from a different national origin realized the phrase refers to high school because she had recently been exposed to Puerto Rican students in her college, two other participants thought “escuela superior” included university education. One of them thought it was the education after elementary school, including high school and university. The other participant thought “escuela superior” was included from grade 6 until earning a bachelor’s degree. In Version 2, the question replaced “escuela superior” with “high school,” which did not prompt these alternate interpretations.
Recommendations
To accommodate survey respondents across diverse educational backgrounds, we recommend a global replacement in the survey of “escuela superior” with “high school.” Furthermore, we recommend including definitions in the help text for GED, HiSET, TASC, and “escuela en el hogar,” should also include the English term homeschooling.
Table 5. N24ADEGREE - Degree program at NPSAS institution
¿Para qué título o certificado estabas estudiando durante tu semestre o trimestre de matrícula más reciente en ________? □ Grado asociado (usualmente es un título de 2 años) □ Título de bachillerato (usualmente es un título de 4 años) □ Título de maestría □ Título de doctorado - investigación/erudición (por ejemplo, PhD, EdD, etc.) □ Título de doctorado en práctica profesional (por ejemplo, en quiropráctica, odontología, leyes, medicina, optometría, farmacia, podiatría o medicina veterinaria) □ Título de doctorado – otro □ Certificado o diploma subgraduado (usualmente toma menos de 2 años), incluyendo programas que permiten obtener una certificación o licencia (por ejemplo, cosmetología) □ Certificado de post bachillerato □ Certificado de post maestría □ Clases a nivel subgraduado □ Clases a nivel graduado |
Note: Only one version of this question was tested.
Overall Question Interpretation
Nine of the 20 participants were enrolled in a bachelor’s degree program during their most recent semester, four were enrolled in an associate degree program; three were enrolled in a master's degree program, two were pursuing 2-year certificates/and licenses, and two were taking undergraduate courses.
Three participants gave different responses in the screener and the actual interview. One person indicated he was enrolled in a bachelor's degree program in the screener, but in the interview reported that he could not answer bachelor’s degree because he did not have his degree yet, and thus chose undergraduate level classes. Another participant indicated graduate level classes in the screener, and in the interview explained that while she had graduated with a bachelor’s degree in 2020, she was taking undergraduate courses to complete prerequisite classes to be eligible for a graduate program. One participant reported enrollment in PhD program during the screener but in the interview reported she was working on her master’s degree.
Several degrees and certificates were unknown terms to participants. Nine participants had trouble understanding “clases a nivel subgraduado” [undergraduate classes]. Other categories participants were unfamiliar with were “certificado o diploma subgraduado” [two-year certificate or diploma], “clases a nivel graduado” [graduate classes], “certificado de post bachillerato” [post-baccalaureate certificate], “certificado de post maestría” [post-master's certificate] and “titulo de doctorado –investigacion/erudición” [doctoral degree – research/scholarship]. The term “erudición” [scholarship] was unfamiliar to one participant. With regards to the doctoral degrees, two participants were not sure about the distinction between the two types of doctoral degrees. One participant also noted that law degrees would be master's degree-level in Puerto Rico, and that in the Dominican Republic “bachillerato” refers to high school; similarly, another participant noted that “licenciatura” was the term for a bachelor’s degree in Mexico.
Recommendations
Terminology about education levels is country dependent. There are no terms that work equally well across the Spanish-speaking world. Decisions about Spanish terminology will need further discussions with RTI and NCES.
Table 6. N24ANENRLFT - Months of full-time attendance at NPSAS institution
Version 1 |
Version 2 |
Primero, nos gustaría recolectar información sobre tu asistencia a tiempo completo.
¿En qué meses has asistido o planeas asistir a <SCHOOL NAME> como estudiante a tiempo completo? No incluyas meses durante los cuales no estés tomando clases, como las vacaciones de verano.
Si has asistido o planeas asistir a <SCHOOL NAME> a tiempo completo durante sólo una parte de un mes, por favor incluye ese mes. Intenta lo mejor que puedas predecir los meses en que asistirás a tiempo completo hasta el 30 de junio de 2023). |
Primero, nos gustaría recolectar información sobre tu asistencia a tiempo completo. ¿En qué meses has asistido o planeas asistir a <SCHOOL NAME> como estudiante de tiempo completo? No incluyas meses durante los cuales no estés tomando clases, como las vacaciones de verano. Si en algún mes has asistido o planeas asistir a <SCHOOL NAME> a tiempo completo durante sólo una parte del mes, por favor incluye ese mes. Intenta lo mejor que puedas predecir en qué meses asistirás a tiempo completo hasta el 30 de junio de 2023). |
Overall Question Interpretation
Fourteen participants were certain about the months they attended, while six said that they were unclear about the months. One participant was not sure because they did not understand the question, and another explained that the uncertainty was due to catching up with coursework and changing her schedule. One participant noted that it was challenging to say what months they attended because classes were constantly cancelled due to strikes and for this participant, attending classes meant being present, not just enrolled.
There was also some misunderstanding about how participants defined “asistencia a tiempo completo” [full-time attendance]. Seven participants defined it as taking a certain number of course credits, anywhere between 9 and 12. Six other participants associated it with taking a certain number of classes, anywhere from two to five classes, although four of these were not sure about the number of classes. Four participants said that full-time attendance was defined by solely attending school and not working, although one of them also said that carrying a 12-credit load counted as full-time. One said being in class for 8 hours constituted full-time attendance.
One person did not understand the second part of the question that mentions attending only part of the month, since they did not know they could attend for only one part of the month. To clarify the ‘part of the month’ concept, Version 2 added “Si en algún mes…” [If in any given month…]. No changes in response patterns or interpretation were detected after this change.
Recommendations
Use Version 2 as it specifies more clearly that the question applies to any month when the participant may have attended partially.
Table 7. N24EPREPAR - Prepare for teaching career at the preK - 12th grade level
Version 1 |
Version 2 |
Tenemos algunas preguntas acerca de cualquier cosa que puedas haber hecho para prepararte para una carrera docente. ¿Has hecho algo para prepararte para una carrera docente desde el nivel de pre-kindergarten hasta el grado 12?
(Por favor incluye solamente preparación formal, como por ejemplo tomar cursos para completar un grado en educación, tomar un examen de certificación, completar un programa alternativo de ingreso aprobado por el estado o completar una práctica de enseñanza.)
|
Tenemos algunas preguntas acerca de cualquier cosa que hayas hecho para prepararte para una carrera en la enseñanza. ¿Has hecho algo para prepararte para una carrera enseñando a estudiantes desde el nivel de pre-kindergarten hasta el grado 12? (Por favor incluye solamente preparación formal, como por ejemplo tomar cursos para completar un grado o título en educación, tomar un examen de certificación, completar un programa alternativo de ingreso aprobado por el estado o completar una práctica de enseñanza.)
|
Overall Question Interpretation
Of the 20 participants, 13 said they had not done anything to prepare for a career in teaching students from pre-K through grade 12; five answered affirmatively. One declined to answer, and another said they had prepared for teaching grades pre-K through grade 4. However, these answers included false negatives among those who answered Version 1.
In probing responses to Version 1, it became clear that the organization of the sentence was generating confusion. A few participants interpreted the question as asking if they had done anything to prepare for a teaching career while they were in pre-K through 12th grade. This, combined with lack of familiarity with “carrera docente” [teaching career], confused participants.
One participant thought a “carrera docente” referred to knowledge in a specific area where one was certified and thought about Excel courses she had taken. One participant did not know what “docente” meant and another associated it with sharing knowledge generally but not exclusively in school. Another participant was confused by the range of grades because they were studying to teach at lower grade levels.
Version 2 replaced “carrera docente” with a synonymous phrase in a lower register: “carrera en la enseñanza.” It also added that students they would teach would be in pre-K through 12th grade. This made the question much easier to interpret and eliminated false negatives.
In both versions, participants were asked to define “práctica de enseñanza.” Three participants associated it with an internship where they could practice what they had learned in class; similarly, one participant thought it referred to practicing what one had learned until mastery, one considered it practice before formal teaching, and another thought of completing hours that correspond to course completion. Another participant more appropriately associated it with doing a lesson plan, and another as being a mentor. One participant said this term was clearer than “carrera docente” because it explicitly mentioned teaching but did not define it, along with a few other participants.
Recommendations
We recommend using Version 2 to eliminate false positives (i.e., incorrect endorsement). It is also recommended that help text for the question include a definition for the term “práctica de enseñanza” given the variation of interpretation.
Table 8. N24EHOWPREP - Steps to prepare for teaching career at the preK - 12th grade level
Version 1 |
Version 2 |
¿Cuál de los siguientes pasos has tomado para prepararte para enseñar desde el nivel de pre-kindergarten hasta grado 12?
|
¿Cuál de los siguientes pasos has tomado para prepararte para enseñar a estudiantes desde el nivel de pre-kindergarten hasta grado 12?
|
Overall Question Interpretation
Nine of the 20 participants did not know what “programas de inducción local” [local induction programs] meant. Two participants offered their understanding of this term as tests to obtain teaching certificates as fast as possible, and as meetings with an expert who teaches teaching methods that are called “inducción magisterial.” Five participants were unfamiliar with “prueba Praxis” [Praxis test], and four had never heard of Teach for America.
Recommendations
It is recommended Version 2 be used and that help text be added with definitions for the unknown terms. Survey designers should develop a definition of “local induction programs” to improve accuracy of translation.
Table 9. N24BFUTWAGES - Future expected yearly wages
Cuando termines toda tu educación, ¿cuál piensas que será tu más probable salario?
$ _____________.00 (por año) |
Note: Only one version of this question was tested.
Overall Question Interpretation
Overall, participants understood the question, and used different strategies to determine their response. These included their previous work experience in their country, specific careers they would like to pursue or are currently studying for, the economic situation of their country, conversations with others and online research, previous job offers, and publicly known university salaries. Others answered based on what they would like to earn. However, two participants were already working following their education, and four did not answer the question. Those that did not respond said that they did not know how to answer, but two of these did provide an estimate based on their research and what they would like to earn.
No changes are recommended for this question.
Table 10. N24CAIDGATE1/ N24CAIDGATE2 - Financial aid gate for the NPSAS academic year
Version 1 |
Version 2 |
Por favor indica si has o no has recibido o usado alguna de las siguientes fuentes para pagar por tu educación.
|
Por favor indica si has o no has recibido o usado alguna de las siguientes fuentes para pagar por tu educación.
|
Note: This question is presented in a radio grid, in which each item is supplied with a ‘Yes’ and ‘No’ radio. In the NPSAS:24 survey, these financial aid gates are administered separately to undergraduate students (N24CAIDGATE1) and graduate students (N24CAIDGATE2) as some items are only applicable at certain levels of education (e.g., fellowships). All response options were displayed regardless of education level to obtain cognitive feedback about all response options.
Overall Question Interpretation
Participants were unfamiliar with several terms in this question, such as “fellowship,” “ayudantía,” and “entrenamiento práctico”. Of the 20 participants, eight could not define these terms at all. Several more had interpretation issues with multiple combinations of these terms and had varying interpretations. Two participants thought “ayudantías” referred to forms of university assistance. One said that “ayudantías” were also known as work-study arrangements. Another reported teaching assistantships were not called “ayudantías” in Puerto Rico, where “ayudantes” are called “asistentes.” For “entrenamiento práctico,” participants thought it was in exchange for a stipend; synonymous with “residencia clinica”, or an internship.
It became apparent in testing Version 1 that participants were unfamiliar with the term “subvenciones” [grants or subsidies]. In addition, the translation for work-study, “estudio y trabajo” was interpreted literally as “studying” [estudio] and “working” [trabajo], rather than as a specific program. For this reason, two changes were made to the translation to improve comprehension. The term “grants” was translated in Version 2 with a synonym, “fondos” [grants]. To clarify “estudio y trabajo”, the English name of the program was added. These two changes were well received and significantly improved comprehension.
Recommendations
We recommend keeping the changes made for Version 2 given the increased understanding. Given the lack of familiarity with several items on the list, we recommend adding help text with definitions of “fellowship,” “ayudantía,” and “entrenamiento práctico.”
Table 11. N24DGIGINTRO - Introduction to nonstandard and gig job section
Version 1 |
Version 2 |
A continuación estamos interesados en obtener información sobre cualquier trabajo por pago no tradicional o gig que hayas tenido mientras estabas en el college, incluso si solamente trabajaste durante vacaciones escolares o cuando no estabas tomando clases. Tendrás la oportunidad de informar sobre otros empleos más adelante. |
A continuación estamos interesados en obtener información sobre cualquier trabajo por pago no tradicional o gig que hayas tenido durante tu tiempo en el college, incluso si solamente trabajaste durante vacaciones escolares o cuando no estabas tomando clases. Tendrás la oportunidad de informar sobre otros empleos más adelante. |
Overall Question Interpretation
Many participants were not familiar with the word “gig.” In addition, a few interpreted the translation of “mientras estabas en el college” [while in college] literally as while on campus rather than about the time when they were in college. For this reason, in Version 2, the translation was changed to “durante tu tiempo en el college” [during your time in college], and this solved the issue.
Recommendations
We recommend Version 2 given improved interpretation of the “while in college” concept.
Table 12. N24DGIGLIST - Paid for any nonstandard or gig job
Version 1 |
Version 2 |
Mientras estabas en el college, ¿te han pagado por alguna de las siguientes actividades?
Por favor descríbelo: _____________ |
Durante tu tiempo en el college, ¿te han pagado por alguna de las siguientes actividades?
Por favor descríbelo: _____________ |
Overall Question Interpretation
Of the 12 participants from Puerto Rico, only four had heard the word “gig” but did not use it. Six participants understood or used the word “chivito” to refer to a nontraditional job, and other two participants mentioned the term “trabajo chiripa”. Seven of the eight participants from the United States had never heard the word “gig” and were not able to provide an equivalent to the word in Spanish; although two of them mentioned the term “un trabajito”.
Participants described “trabajo no tradicional” [nonstandard job] in a variety of ways, such as jobs that did not have a regular schedule, jobs that might not provide a steady income or where employees get paid cash, and jobs where the employee had the ability to control the number of hours and the days they worked.
Two people considered “trabajo no tradicional” [non-traditional job] as not related to a person’s studies or professional preparation: “Sporadic or seasonal jobs unrelated to your profession” and “A job that you do outside your study time or has no relation to your career or professional preparation.”
Recommendations
We recommend utilizing Version 2 that carried through the improvement of the “while in college” concept from N24DGIGINTRO Version 2. Additionally, we recommend keeping “trabajo no tradicional” and deleting “o gig” from the “Algún otro” [some other] response option, because terms that participants volunteered are very informal and depend on country of origin.
Table 13. N24HINCOM - Income for prior calendar year from all sources
¿Cuál fue tu ingreso en el año natural 2022, antes de impuestos y deducciones?
(El año natural 2022 va del primero de enero de 2022 al 31 de diciembre de 2022. Incluye todos los ingresos por los que pagaste impuestos, incluyendo trabajo, ingresos por inversiones o pensión alimenticia. No incluyas ninguna subvención o préstamo que hayas usado para pagar tus estudios, ni tampoco dinero que tú familia te haya dado).
|
Note: Only one version of this question was tested.
Overall Question Interpretation
Most participants understood this question clearly and provided an answer. Two participants, however, did not know what income they had to include in their answer. One student asked if she could include the money she received from her family and from her university. She explained that she was getting paid for studying in Puerto Rico, but that those funds were kept in her home country (not U.S.) to pay the salary of the person who was substituting her in her teaching job. Another participant who personally only had work-study income, answered about her household income. Three participants mentioned they had not known the term “año natural” [calendar year], but that they understood the definition provided.
Recommendations
No changes recommended.
Table 14. N24EFIN2000 - Confidence could come up with $2,000 within next month
¿Qué tan seguro(a) estás de que podrías obtener $2,000, de cualquier fuente disponible, si surgiera una necesidad inesperada dentro del próximo mes? ¿Tú...
|
Note: Only one version of this question was tested.
Overall Question Interpretation
Most participants were able to answer this question without problems; however, one person did not understand what “de cualquier fuente disponible” [from any available source] meant. This participant was able to answer the question by replacing “de cualquier fuente disponible” with “de donde sea” [from wherever]. Another participant was expecting that the question would provide more information on what that “fuente” [source] would be. This participant answered “Probably come up with the $2,000” because he was an optimistic person, not because he was able to think about a specific source.
While answering this question, participants thought about available sources such as their own income from work, their own savings, credit cards, loans, and help from family and friends. One participant also mentioned the American Rescue Plan Act of 2021, and thought she could receive financial help from the emergency aid grant because she was a low-income student.
Recommendations
No changes recommended.
Table 15. N24FSX – Sex assigned at birth
Las siguientes preguntas nos ayudarán a entender mejor las experiencias de personas de todo tipo de orientación sexual y de identidad de género.
¿Qué sexo te asignaron al nacer en tu certificado de nacimiento original?
|
Note: Only one version of this question was tested.
Overall Question Interpretation
Of the 20 participants only one had a comment about this question. She was able to provide an answer but said that she was confused because, for her, in the context of the question, “sex” referred to “man/woman” and not “male/female.”
Recommendations
No changes recommended given the level of comprehension by most respondents. The question asks about “sexo,” to which “man/woman” are not a possible answer, “sexo” can only be qualified by “male” or “female” in Spanish, which aligns with the English.
Table 16. N24FGENDERID - Gender identity
¿Cuál es tu género? Tu género es cómo te sientes por dentro y puede ser el mismo o diferente de tu sexo biológico o de nacimiento.
Por favor descríbela:____________
|
Note: Only one version of this question was tested. In the NPSAS:24 field test survey, the item “Dos espíritus” [Two-Spirit] only displayed to American Indian/Alaskan Native respondents. All response options were displayed regardless of race to obtain cognitive feedback about all response options.
Overall Question Interpretation
All participants provided an answer and did not exhibit any discomfort with the question. All but two participants reported that they had never heard of the term “dos espíritus” [Two-Spirit], while three others incorrectly related it to someone who identifies as bisexual because of the word “two.” Three participants had never heard the term “no binario” [nonbinary]. Those who had heard the term described it as “people who do not identify with the pronouns he or she” or “people who do not identify with the terms man and woman.” One participant defined “transgénero” as gay.
Recommendations
There are currently several sizable federal studies looking into the best way to ask SOGI questions in Spanish through cognitive testing. From our work so far in these projects, we are starting to see that unacculturated Spanish-speaking immigrants do not share the same classification of sexual orientation versus gender identity and are not familiar with many terms English speakers use. For some, as we saw in the NPSAS testing, being a sexual/gender minority is not classified into either sexual orientation or gender identity, but as something other than straight and cisgender. We recommend keeping the question as is. The stem and definition were clearly understood and interpreted as intended, and everyone found a response that fit their situation.
Table 17. N24FLGBTQ - Sexual orientation
¿Cuál de las siguientes opciones representa mejor cómo te identificas?
Por favor descríbela _____________
|
Note: Only one version of this question was tested. In the NPSAS:24 field test survey, the item “Dos espíritus” [Two-Spirit] only displayed to American Indian/Alaskan Native respondents. All response options were displayed regardless of race to obtain cognitive feedback about all response options.
Overall Question Interpretation
Of the 20 participants, only one person did not provide an answer, because he was not sure what several of the response terms meant. He had never heard the term “heterosexual” or any of the other terms except “lesbiana o gay” [lesbian or gay]. Another participant thought that this was the same question as N24FGENDERID and had to read it again to see the difference and provide an answer.
Three participants were not familiar with the term “asexual,” six were not familiar with the term “pansexual,’ eight were not familiar with the term “queer,” and one was not familiar with the term “bisexual.”
Recommendations
The stem was clearly understood and interpreted as intended, and everyone found a response that fit their situation, except for one respondent. Because this respondent represented a common situation, as described in the literature and in our experience, we recommend replacing ‘heterosexual’ with ‘No gay ni lesbiana, es decir, heterosexual’ [Not gay or lesbian, that is heterosexual], a formulation that has been found to reduce nonresponse to this question1.
Table 18. N24BEQUITABLE - Emphasis on equitable treatment at NPSAS institution
Por favor usa un número del 1 al 5 para responder a las siguientes afirmaciones. El número 1 significa “completamente en desacuerdo” y el 5 significa “completamente de acuerdo”.
Tomando en cuenta tus experiencias mientras asistías, <SCHOOL NAME>...
1 (Completamente en desacuerdo) 2 (Algo en desacuerdo) 3 (Ni en desacuerdo ni de acuerdo) 4 (Algo de acuerdo) 5 (Completamente de acuerdo) |
Note: Only one version of this question was tested.
Overall Question Interpretation
Most participants were able to select one of the answer options provided; however, in probing it became evident that their narrative did not always match their answers. Additionally, some participants were not able to provide answers, and some reversed the Likert scale while answering.
For the statement: “Proporcionó a los estudiantes los recursos necesarios para triunfar en un mundo multicultural.” [Provided students with the resources needed for success in a multicultural world], one student who had initially answered “Completely agree” changed her answer to “Neither disagree nor agree” during probing because, as she explained, it was not the university who provided these resources. “I see this initiative more on the part of the university community itself, perhaps not coming from activities created directly from the administration. I would see that student groups were the ones coming together to promote this.” Two participants who also answered “Neither disagree nor agree” mentioned that they had not experienced anything that could help support a different answer. Another participant answered “Neither disagree nor agree” because she did not understand what “multicultural” meant in the statement. She explained that at the university she attended everyone was from Puerto Rico except for a student who came from another country. She felt that the word “multicultural” would not apply to that university.
For the statement: “Se aseguró que los estudiantes no fueran discriminados debido a sus identidades, por ejemplo, raza/grupo étnico, género, religión, orientación sexual, etc.” [Ensured that students were not stigmatized because of their identity (e.g., racial/ethnic, gender, religious, sexual orientation, etc.], a participant did not know what to answer since he received asynchronous virtual classes that consisted of recorded lessons, had no contact with the other students, and little contact with teachers. One person who answered, “Completely disagree” reversed the scale, she meant to answer, “Completely agree.” Some participants who answered, “Completely agree” mentioned reasons such as “they always had discrimination policies included in materials, activities and in everything” and “the university, from my experience, has many programs that really provide guidance on this subject.”
For the statement: “Tomó en serio las acusaciones de discriminación o acoso.” [Took allegations of discrimination or harassment seriously.], six participants who had not experienced any discrimination or harassment or knew of anyone who had experienced it, answered “Neither disagree nor agree.” However, there were others who did not provide an answer or provided an answer other than “Neither disagree nor agree.” For example, two participants answered “Somewhat agree” and two others answered “Completely agree” also without having experience, because they assumed that the school would take accusations of discrimination or harassment seriously. Two participants who knew of someone who had experienced discrimination or harassment, answered “Somewhat disagree.” Another participant who had experienced discrimination or harassment herself answered “Completely disagree.” In all three cases the participants explained that the school had not dealt with the accusations as expected. Lastly, one participant who had originally answered “Neither disagree nor agree” paying attention to only the word “discrimination” changed her answer to “Completely disagree.” She explained that in terms of “acoso” [harassment] she knew of cases where teachers had been reported to the corresponding administrative authorities, but these teachers had not been removed from their positions.
For the statement: “Ayudó a los estudiantes a desarrollar habilidades para enfrentar la discriminación y el acoso.” [Helped students develop the skills to confront discrimination and harassment.], two people who answered “Completely disagree” explained that the school had never provided lectures or materials related to dealing with discrimination and harassment. A third participant who had experienced both discrimination and harassment, explained that the school did not take her complaints seriously. This participant stopped going to school for months because the school did not provide resources to deal with these issues. Some of the participants who answered “Completely agree” mentioned things such as events, programs, and informative lectures offered by the schools to help students develop the skills to confront discrimination and harassment.
Recommendations
Consider reversing the Likert scale. Although there are sound methodological reasons for going from disagree to agree in this question, reversing the scale may reduce response error. Adding a choice for “Did not experience this” or “Don’t know” may be useful to prevent midpoint answers that reflect uncertainty.
Table 19. N24BVALUED - Value and belonging at NPSAS institution
Por favor usa un número del 1 al 5 para responder a las siguientes afirmaciones. El número 1 significa “completamente en desacuerdo” y el 5 significa “completamente de acuerdo”.
Cuando asistía a <SCHOOL NAME>…
1 (Completamente en desacuerdo) 2 (Algo en desacuerdo) 3 (Ni en desacuerdo ni de acuerdo) 4 (Algo de acuerdo) 5 (Completamente de acuerdo) |
Note: Only one version of this question was tested.
Overall Question Interpretation
Most participants were able to select one of the answer options provided for each statement without difficulty, and their explanation for choosing their answers matched their response.
For the statement: “Me sentía valorado(a) como persona.” [I felt valued as an individual.], one of the participants who answered “Somewhat disagree” explained that her answer had to do with the complicated administrative processes she had to go through when enrolling in school.
For the statement: “Sentía que pertenecía allí.” [I felt I belonged.], one participant who answered ‘Somewhat disagree” explained that the university had closed the campus residence and that was something that helped create a sense of belonging for the students, which is no longer there.
For the statement: “Pensé en dejar [NPSAS INSTITUTION] porque me sentía aislado(a) o no me sentía bienvenido(a).” [I considered leaving [NPSAS INSTITUTION] because I felt isolated or unwelcomed.], no interpretation issues were detected.
For the statement: “Me trataban con respeto.” [I was treated with respect.], one person answered “Neither agree nor disagree” because there were people in the university who had treated him with respect and others who had no empathy and had not treated him with respect.
For the statement: “Sentía que otros no valoraban mis opiniones.” [I felt others didn’t value my opinions.], one participant who answered “Somewhat agree” explained her answer: “Because I’m from another country, I have a different accent, I'm different. It is not that they didn’t value me, it is just that I had to show more who I am so they would value me. After a while they already knew me.” And another participant who also answered “Somewhat agree” explained that classes were very large and sometimes she felt she was not being heard.
For the statement: “Tenía oportunidades de éxito académico similares a las de mis compañeros.” [I had opportunities for academic success that were similar to those of my peers.], a participant answered, “Somewhat disagree.” This participant explained that she felt that teachers were more interested in students with good grades and who participated more in class, and she was not one of those students. Another person answered, “Neither disagree nor agree” and explained that although everyone was treated equally, he felt not everyone had the same opportunities because not everyone had the same economic level.
For the statement: “Encontré una o más comunidades o grupos a los que sentía que pertenecía.” [I found one or more communities or groups where I felt I belonged.], one participant did not provide an answer. This participant explained that she felt that as an older student she did not feel there were communities or groups to which she could belong. Another person who answered, “Completely disagree” simply said: “I never joined a group.” And another participant (1042) who also answered “Completely disagree” explained: “I am a solitary person; it is hard for me to let people know what I don’t understand. I’m not one to say I don’t understand this, and obviously I didn’t seek the help I needed, and what I did is that I isolated myself and always stayed apart.”
For the statement: “Tenía que trabajar más duro que otros para que me valoraran de igual manera.” [I had to work harder than others to be valued equally.], three of the four participants who answered, “Completely agree” said that because of their limited English language skills they sometimes felt that they had to work harder than others who spoke English, not necessarily to “be valued equally,” but to learn in general. They explained that it took them more time and effort to really grasp the content of what they learned in class, whereas other students did not have that issue.
No changes recommended.
Table 20. N24BRACECOMP - Racial composition of pre-NPSAS community
La siguiente pregunta es sobre tus experiencias antes de asistir a <SCHOOL NAME>
Piensa en la comunidad en que creciste. ¿Cuántas otras personas…
3 (Todas) 2 (La mayoría) 1 (Algunas) 0 (Ninguna) |
Note: Only one version of this question was tested.
Overall Question Interpretation
These items elicited varied interpretations. One person thought that “comunidad” could refer to many things, including the community where she lived, academic, or religious community. For one participant “comunidad” was the general city she grew up in and another thought about town, economy, or school. Others thought about country of origin.
The reference in the question to the community “where you grew up” [comunidad en que creciste] caused some confusion. A participant thought that this question could be made clearer if it specified “cuando eras niña” [when you were a child]. Another person answered about her time in elementary school, since this was when she spent the most time when she was young. Similarly, another participant did not think the question referred to her childhood. A participant also said that this part of the question was unclear in terms of age, whether it referred to early childhood or adolescence. Three others answered with regards to their university experiences, not earlier ones as the question intended. Another participant thought this question was referring to where they lived at present, and some thought about high school.
There was variation in how participants interpreted “se veían como tú,” [looked like you]. Several participants understood this to refer to physical appearance, one participant specified skin tone. However, other participants thought this could mean things other than physical appearance, including socioeconomic status, and behavior. One person suggested that the question read “se veían físicamente como tú” [look physically like you]. Another this was a broad concept and did not think it referred to physical appearance because they could not see how this was related to school.
Recommendations
We recommend anchoring the timeframe to a specific age or school level, for specificity and to reduce lack of clarity. Additionally, if “looked like you” is meant to be about observable physical appearance, this should be clarified.
Table 21. N24HPARED1 - Parent 1: highest level of education completed
Tenemos algunas preguntas sobre el nivel más alto de estudios que tus padres completaron.
Por favor selecciona a uno de tus padres:
|
Note: Only one version of this question was tested.
Overall Question Interpretation
Most participants understood this question clearly. However, some were not certain which option best described their parents’ education level. One participant did not notice the question asking for the highest level completed, and she selected multiple answers2. Several participants noted that their parents’ education level was not listed (i.e., post-baccalaureate and medical doctor). Additionally, several participants’ parents did not complete elementary school or middle school. Of these, one person whose parents completed middle school selected “no terminó la escuela superior,” and another remarked that both parents only completed the first grade of elementary school and did not select an option. One participant whose mother did not complete elementary school did not select an option.
Recommendations
As mentioned earlier, we recommend the global replacement of “escuela superior” with high school. No other recommendations given the high level of comprehension.
Table 22. N24HPARST - Parents’/guardians’ marital status
¿Cuál es el estado civil de tus padres (o tutores)? Si tus padres (o tutores) están divorciados, por favor responde esta pregunta sobre el estado civil del padre (o tutor) con quien viviste más tiempo durante los últimos 12 meses.
(Si no viviste con uno de tus padres más tiempo que con el otro, responde tomando en cuenta al padre que proveyó más sostén económico durante los últimos 12 meses o durante el año más reciente en el que recibiste sostén económico de uno de tus padres).
|
Note: Only one version of this question was tested.
Overall Question Interpretation
This question was cognitively burdensome to some participants. One participant asked to have the question re-read. Another person specified that the information in the parenthetical was particularly confusing, and one suggested that the question could be made shorter. Another participant said the question had a lot of information and that it was necessary to read it carefully.
The reference to the respondent living with parents caused confusion among a few participants. This was the case for participants who no longer lived with their parents. For example, one was surprised by the reference to the parent she lived with the most in the past 12 months and said as an adult the question sounded ridiculous. Similarly, one participant indicated not living with their parents anymore. Another one had difficulty understanding the concept of living with a parent longer than with the other, as her parents lived together. However, most of this confusion results from all cognitive participants receiving this question, regardless of their dependency status. For the NPSAS:24 student survey, N24HPARST is only administered to respondents who are dependent students (i.e., younger than 24 years old, unmarried, had no dependent children, and were not a veteran or on active duty in the military).
There were also instances in which multiple options applied to participants’ parental situations. For example, one participant reported their parents were divorced and both deceased.
Recommendations
The question can be improved if the parenthetical instructions that follow the question itself were included in help text to reduce the amount of content on screen.
Overall, testing proceeded without major difficulties. Most participants focused on the task at hand and provided their best answers and rationales for their responses. They provided rich narratives that allowed us to understand interpretation patterns and response error.
As discussed in this report, there were interpretation issues in some of the questions tested. Several items required adding further context or use of different words to clarify the scope of the question. Nine questions were modified to include revised wording at the midpoint of data collection. Upon analysis, we found that these modifications aided in comprehension and generally recommended their adoption.
In addition to comprehension issues, respondents were also unfamiliar with several terms and lacked familiarity with some concepts (e.g., different types of paid work helped to finance education, the English term ‘gig’ was unknown to participants). In other cases, comprehension issues were due to variation in terminology by country of origin. For example, in the question N24ADIPL, we recommended that the full-scale study replace “escuela superior” with “high school,” because this was a term used in Puerto Rico as high school, whereas it is not used elsewhere. This also came up in the N24ADEGREE question in which certain educational levels had different meanings across national contexts, such as “bachillerato.” Many other educational terms were completely foreign to participants, evident in questions about degrees in N24ADEGREE, as well as terms about teaching training in N24EHOWPREP, and terms around financial aid in N24CAIDGATE1/2. We recommended adding help text with definitions of these terms that respondents could consult if desired.
With regards to sexual orientation and gender identity questions, several participants were unfamiliar with response options that English speakers use, although we found that respondents were able to find a response that fit their situation. As such, we recommended keeping the questions as they were. For the sexual orientation question, however, we did recommend a change to reduce non-response as recommended in the literature: to reverse terms in the heterosexual response option and use “Ni gay ni lesbiana, es decir, heterosexual.”
1 See Michaels, Stuart et al. 2017: https://pubmed.ncbi.nlm.nih.gov/29154714/
2 However, for the NPSAS:24 student survey, this question is a radio button list in which only one selection is possible.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2024-07-20 |