Responses to OMB questions

Responses to OMB passback questions for NAEP Wave 3 submittal

1. Please work with Marilyn Seastrom to harmonize the confidentiality pledges across instruments.  Right now, there are two distinct pledges within the same collection.

The OMB package erroneously contained two versions of the pledge text. Beginning with the 2008 NAEP assessments, we were advised (by Marilyn Seastrom) on which text to use on the student, teacher, and school questionnaires. Prior to that, different text was used for the teacher and school questionnaires. All final production versions of the questionnaires used the correct text in 2008 and the 2009 questionnaires will use that same text.

The correct confidentiality text is “A project of the Institute of Education Sciences.

This report is authorized by law (P.L.107-110, 20 U.S.C. §9010). While your participation is voluntary, your cooperation is needed to make the results of the survey comprehensive, accurate, and timely. The information you provide is being collected for research purposes only and will be kept strictly confidential.

OMB No. 1850-0790 Approval Expires 05/31/2010

2. Please provide more information about the membership of the Background Variables Standing Committee.

The committee is composed of leading experts in sociology, economics, psychology, and demography, most of whom have had some prior experience with NAEP. In addition to the regular committee members, there are 6 liaison members who additionally serve on a content committee (e.g., Reading, Mathematics, Civics, and Science).

3. Please provide more information about the "field testing to accommodate six-month reporting" and why it may no longer be needed.

Field testing was introduced in 2002 for the subjects that require six-month reporting (i.e., mathematics and reading at grades 4 and 8). This step enabled some of the analyses to be conducted prior to the condensed timeframe required to meet six-month reporting.  Given increased computing power, software program improvements and enhancements, automated quality control checks, and the power and stability that resulted from the larger samples now obtained for assessments in these subject areas, the additional step of field testing is no longer required.

4. Is the supplemental information collected on ELL and disabled students regardless of whether they participate in the NAEP assessment?

Yes, the appropriate SD and/or ELL (supplemental) questionnaire is completed for any student confirmed as a Student with Disabilities (SD) and/or an English language learner (ELL), regardless of whether or not the student takes the NAEP assessment.

5. Please clarify the rationale for why NCES collects transcript data without a student or parent consent process.

The collection of student transcripts as part of the national High School Transcript Study (HSTS) is covered under provisions of the Family Education Rights and Privacy Act (FERPA) (20 U.S.C. 2343g), as implemented by 34 CFR 99.31 (a)(3)(ii) and 99.35. These laws and regulations permit an educational agency to disclose records to authorized representatives of the Secretary of Education without the prior consent of survey participants, in connection with the audit and evaluation of Federal and State supported education programs. The privacy of the information will be protected as required by FERPA.  As part of the study, a Disclosure Notice is placed in each eligible student's file.  One side of the notice explains Westat’s authority to conduct the study; the reverse side presents the appropriate FERPA provisions regarding the conditions under which the disclosure of student records may be made without prior consent.   However, if a school requires consent, the contractor follows the requirements of the school for consent.  

6. Please clarify whether non-AI/AN students are included in the sample for the NIES study.  If so, do they receive the same background instrument?

Only students who are identified as AI/AN are included in the sample for the NIES study and receive the NIES questionnaire. 

7. Please provide specific justification for the collection of SES measures on the NAEP.

Under the No Child Left Behind Act of 2001 (Public Law 107-110), NAEP is required to collect information on and report achievement results disaggregated by socioeconomic status (SES), as stated as follows (U.S. Department of Education, 2002):

“Results of the assessments are reported for the nation and states in terms of average scores as well as the percentage of students that reach each of the Governing Board’s three achievement levels: basic, proficient, and advanced. NAEP results will be disaggregated to the extent feasible by race, ethnicity, gender, socioeconomic status, disability, and limited English proficiency.” (GENERAL PROVISIONS, NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS, TITLE VI, SUBPART C, NATIONAL EDUCATION STATISTICS ACT, SECTION 411)

U.S. Department of Education, Office of Elementary and Secondary Education, No Child Left Behind: A Desktop Reference, Washington, D.C., 20202. (Retrieved from www.ed.gov/admins/lead/account/nclbreference/ August 28, 2008).

8. Please provide a summary of and specific literature citations in support of the SES proxy variables being proposed.  This should include information about age-appropriateness for the measures, as well as their validity absent an ability to adjust for household size. 

SES is generally considered a composite of family income and wealth, parental education, and parental occupational status (Hauser & Warren, 1997). The SES proxy variables proposed for inclusion on the background questionnaire can be categorized by these components--parental educational attainment, parental occupational status, financial resources—along with household composition.

Parental Educational Attainment

Parental educational attainment is typically identified in one of two ways: (1) by asking how many years of education parents have completed or (2) by asking the highest degree parents have attained (allowing options for parents who pursued but did not complete a degree). Research has found that the highest-degree formulation tends to have a stronger association with other measures of social and economic standing, particularly with regard to income and occupational prestige (Smith, 1994). Although we are able to collect this information from 8th- and 12th-grade students through the background questionnaire, cognitive laboratory studies and NAEP field test data have shown that fourth-graders have difficulty with questions about parental educational attainment. For this reason, NAEP does not collect these data at the fourth-grade level.

Occupational/Employment Status

Contemporary research highlights the relationship between socioeconomic status and occupation (Ganzeboom & Trieman, 1996). Entry into specific occupations is strongly affected by educational qualifications), while employment is the major source of income. Thus, occupational positions are strongly tied to education and income (Hauser, 1994). In previous cognitive laboratory research, NCES attempted to develop a suitable measure of occupational status (based on Stricker, 1988), but found students experienced difficulty in locating their parents’ specific occupations and reported that the task required a burdensome amount of reading (Hauser, 1994). For this reason, NCES proposed a question on parent employment status.

Financial Resources

NAEP currently asks 4th-, 8th-, and 12th -grade students five questions about access to educational resources at home (i.e., to a newspaper, to magazines, to books, to a computer, and to an encyclopedia). Because these educational resources are indirect and weak measures of financial resources, additional items to measure family income and wealth were proposed as a part of this pilot study. Monetary resources provide a direct means of acquiring goods and services, thus social scientists have devised numerous indicators of financial means (Krieger, Williams, & Moss, 1997). Related to income is the concept of wealth, which is defined as “accumulated assets, typically accrued through inheritance, investment, and other forms of savings” (Krieger et al., 1997). Historically, most families’ primary residence accounts for the largest proportion of their wealth (Keister & Moller, 2001; Spilerman, 2000), which is the justification for the question about renting or owning the home.

Household Composition

NCES proposes asking students about the composition of their households (i.e., family structure and number of siblings) for two reasons. First, household composition is strongly correlated with socioeconomic status (Acock & Kiecolt, 1989; Biblarz, Raftery, & Bucur, 1997; McLanahan, 1985) and student achievement. Second, respondents in Grades 4, 8, and 12 are likely to be able to answer such questions accurately (US DOE: NCES, 1997). This recommendation is consistent with NAGB’s Background Information Framework for NAEP, which states that number of siblings and family status “may be of considerable importance in constructing an SES index” (p. 26). NCES proposes including questions on the following household composition topics: family structure (i.e., presence in the home of adults and other children) and number of siblings.

For these questions, age appropriateness was initially tested via cognitive labs, and then later in the 2007 pilot study. Data from both sources indicate that the specific wording of these questions was accessible and understood by students at the targeted grade levels. Household size is asked for in the questionnaires (all grades), which permits analysis of the structure of relationships between the SES proxy variables and household size. However, the research evidence for the validity of the proxy variables is not dependent on the effects of household size.

References

Acock, A.C. & Kiecolt, K.J. (1989). Is it Family Structure or Socioeconomic Status? Family Structure during Adolescence and Adult Adjustment, Social Forces, 68(2), 553–571.

Biblarz, T.J., Raftery, A.E., & Bucur, A. (1997). Family Structure and Social Mobility. Social Forces, 75(4), 1319– 1341.

Duncan, G.J., & Petersen, E. (2001). The long and short of asking questions about income, wealth, and labor supply. Social Science Research, 30(2), 248–263.

Ganzeboom, H. B.G. & Treiman, D.J. (1996). Internationally Comparable Measures of Occupation Status for the 1988 International Standard Classification. Social Science Research, 25, 201–39.

Hauser, R. M. & Warren, J.R., (1997). "Socioeconomic Indexes for Occupations: A Review, Update, and Critique." Pp. 177-298 in Sociological Methodology 1997, edited by Adrian E. Raftery. Cambridge: Basil Blackwell.

Hauser, R. M. (1994). Measuring Socioeconomic Status in Studies of Child Development. Child Development, 65, 1541–45 and Adams, R. & Wu, M. (Eds.). (2002). PISA 2000 Technical Report. Paris: Organisation for Economic Co-operation and Development.

Keister, L.A. & Moller, S. (2000). Wealth Inequality in the United States. Annual Review of Sociology, 26, 63–81.

Krieger, N.D., Williams, R. & Moss, N.E. (1997). Measuring Social Class in US Public Health Research: Concepts, Methodologies, and Guidelines. Annual Review of Public Health, 183, 41–78.

McLanahan, S. (1985). Family Structure and the Reproduction of Poverty. The American Journal of Sociology, 90(4), 873–901.

Smith, T. (1994). Some Aspects of Measuring Education. General Social Survey Methodological Reports: Methodological Report 83. Chicago: The National Opinion Research Center.

Spilerman, S. (2000) Wealth and Stratification Processes. Annual Review of Sociology, 26, 497–524.

Stricker, L. J. (1988). Measuring Social Status with Occupational Information: A Simple Method, Journal of Applied Social Psychology, 18(5), 423–437.

U.S. Department of Education. National Center for Education Statistics. (1997). NELS: 88 Survey Item Evaluation Report. (National Center for Education Statistics No. 97-052). Retrieved on February 15, 2006, from http://www.nces.ed.gov/pubs97/97052.pdf .

9. Please also describe whether and how NCES plans to link these data to external variables, such as urbanicity, to assist in interpreting them appropriately.

NCES does not plan to link data to external variables such as urbanicity as a primary strategy to help in the interpretation of SES proxy variables. However, in the course of validating the pilot data some analyses will compare these data to external data (i.e., Census data).

10. To what degree has the NAEP group collaborated with other parts of NCES on the development and use of proxy-SES measures?

The NAEP group consulted with and reviewed questionnaires from all NCES activities that collected data from students on socio-economic status variables. Other NCES groups are aware of this study and are interested in NAEP’s findings and potential implication for other surveys.

11. What is the evidence from NAEP pilot studies that there are schools who will report no electricity in their science labs (Part IV, Science)?

Based on the 2008 pilot (School Questionnaire, Grade 4, Part IV, Question 4d), 10.5% of fourth-grade students had school administrators who responded that they did not have electricity in their school’s science laboratories. For the eighth-grade version of this question, the percentage was 2.2%.

12. Why do the instructions vary on the race questions used between the NIES and the other instruments?

The first of the two race/ethnicity questions with instructions in the 4th and 8th grade NIES teacher and school questionnaires should have been the same as the question used in other instruments and thus the question on the relevant questionnaires will be replaced.

13. Typos etc:

    1. Pages 12-14 of the supporting statement have some formatting problems, especially the table on pages 13-14, which makes it difficult to read.

    2. The sentence immediately preceding table 3 says "teacher questionnaires," but should read "school questionnaires."

    3. 4^th grade teacher questionnaires, part II, instructions (and elsewhere)-Since there is another skip instrument at the start of Parts III and IV, suggest deleting "or Part IV (science)" in the following sentence: "If you do not teach reading, English, or language arts, please skip to Part III (mathematics) or Part IV (science)."

    4. 4^th grade pilot teacher questionnaire item 13 (and elsewhere), we think "their" should be "its" in "percentage of students in your mathematics class is completing their assignments?"

 

13a and b. Volume 1 will be revised to correct the sentence preceding the school table on page 15 (change ‘teacher’ to ‘school’) and correct some formatting problems on pages 12-14.

13 c. We will revise the skip pattern direction as suggested.

13 d. Editors concurred with original wording, stating that ‘their’ modifies students, thus the plural is correct.

Two additional response requests

1. Please provide some background on the purpose and specific research objectives of the NIES study, including the purpose of questions asking about specific types of activities such as field trips.

The National Indian Education Study (NIES) includes an examination of the performance of American Indian or Alaska Native students on the National Assessment of Educational Progress (NAEP), and a survey of various factors that might affect the educational achievement of this population. The survey asks about American Indian and Alaska Native students’ educational experiences, their teachers’ educational practices, and school related factors from the school administrators.

 

Questions about specific activities, such as field trips, mathematics related activities, and use of American Indian and Alaska Native languages are designed to inform researchers about the level of exposure to cultural activities.  From the directions given to field researchers, the stated purpose is simply to determine if the students are being given the opportunity to engage in these types of activities.  The history and traditions of American Indian and Alaska Native peoples are the primary focus of these field trips.  The underlying assumption is that these types of activities provide additional value beyond regular classroom activities, and that because they are tied closely to the culture, traditions and history of the AI/AN people, it is of particular importance to ask these questions.

 

2. To the extent that NAEP is piloting some potential new measures of SES, please explain how SES is currently measured in NAEP, as well as how reliable NCES considers those measures. If the measures are based on eligibility for free- and reduced-price lunches, please explain the methodology for how those data are currently obtained and incorporated into the results.

The primary SES proxy variable used for reporting is the income proxy, eligibility for free- and reduced-price lunch (National School Lunch Program, NSLP). It is obtained from school records. In NAEP reports, achievement scores are presented separately for students who are eligible for a free lunch; eligible for a reduced-price lunch, and not eligible. There have been some questions about the validity of the NSLP variable because of students' and students' families’ inconsistency in applying for the program. As an example, among Black and Hispanic students, 70 percent of 4th-graders were reported eligible; 60 percent of 8th-graders were reported eligible, and just over 40 percent of 12th-graders were reported eligible. This sharp decline is not credible because although a modest decline in poverty with age may be seen in Census data, it is generally not this large.

7