Response to OMB Passback

MEMORANDUM OMB # 1850-NEW v.1

DATE: October 30, 2013

TO: Shelly Martinez

Office of Information and Regulatory Affairs, Office of Management and Budget

FROM: Dana Kelly

National Center for Education Statistics

THROUGH: Kashka Kubzdela

National Center for Education Statistics

SUBJECT: Response to OMB passback on PISA 2012 Validation Study

1. Is there a precedence for texting as contact method for minors?  I'd want to be assured that this does not raise novel consent or other similar types of policy issues, and methodologically, I'd prefer to either introduce it for the first time as an experiment or some other carefully monitored way.

At the end of Part B, section B.2.1, we added the following text:

The National Health and Nutrition Examination Survey (NHANES) has been using texting as a mode of contact with minors aged 12 to 16 for two years. NHANES is a household survey that includes an interview and a medical screening. The target is to complete 5,000 examined participants per year. A subsample of persons are preselected for morning exams requiring participants 12 years and older to fast. The study is always trying to improve fasting compliance. In May 2011, NHANES added text messages for the morning appointments that require fasting, after which the NHANES fasting rate went up 2 percent. While NHANES cannot say the text messaging was a direct cause for increasing or helping fasting rates, as those giving permission to text may be a more compliant group, they have seen the fasting rates remain steady, found no evidence that texting negatively impacts rates, and while being inexpensive it may be helping to increase compliance. To implement the NHANES text message reminders, questions were added to the household questionnaire asking permission to text participants along with an explanation of possible fees and asking for the cell phone number of the participant if it has not already been provided. The PISA Validation Study plans to use a similar procedure in ascertaining whether to send text messages to participants.

2. Is a 90% response rate on first contact reasonable? What would that cost?

At the end of Part B, section B.1.1, we added the following text:

A total of 6,116 students were assessed in the national administration of PISA in 2012. Of these, approximately 5,810 students (95 percent) completed a Student Information Form. Among those, 1,081 were assessed in financial literacy, leaving 4,729 students to serve as the starting sample for the main PISA Validation Study.

We expect we will be able to locate 90 percent of the 5,810 students from the national PISA 2012 sample who took PISA mathematics, reading, or science assessments in fall 2012 and completed the Student Information Form providing us their contact information. Locating 90 percent of those students a year later includes responses to address updates from a portion of the sample, but also relies on intensive tracing, as described in the supporting statements, which will use cost-effective tracing techniques, such as calling directory assistance, calling contacts provided by the respondent, and using publicly available Internet searches. In line with other studies that employ locating of participants such as High School Longitudinal Study (HSLS) and Beginning Postsecondary Students (BPS) Longitudinal Study, we estimate that between 15 to 20 percent of the 5,810 students will respond to the first follow-up request for address verification, and the remainder will have to be located using tracing techniques. However, the majority of these students will still be in high school and should be relatively easy to locate. A limited number of schools may be contacted to request assistance in locating students who could not be located using all other methods.

It is expected that there will be a 10 percent loss of sample each year, resulting in 3,447 students for the 2015 study. The 1,081students assessed in financial literacy will be traced and then a subsample of 200 students will be selected for the field test. Approximately 306 of the 6,116 students did not complete the Student Information Form. NCES will not pursue these students as part of this survey; by not completing the form they have implied that they do not wish to be contacted for future surveys.

3. OMB will consult internally as to whether there anything really objectionable about the subjective well-being measures and other BQ modules. When NCES sees the final modules and how they are used in the conditioning module, NCES will recommend whether and how to use the modules. Brian and NCES commented extensively on the OCED's draft of the recently released "Guidelines for Measuring Subjective Well-Being." So we would like confirmation now that the list of topics you have submitted tracks well with those guidelines, and of course assurance that the questionnaire module does once finalized, should NCES decide it would like to include it in this study.

At the end of Part A, section A.1, we added the following text:

In preparation for collecting information on respondent’s subjective well-being we will consult the OECD’s guidelines for measuring subjective well-being (http://www.oecd.org/statistics/Guidelines%20on%20Measuring%20Subjective%20Well-being.pdf). We will use the guidelines when defining specific research questions and then evaluating the identified outputs against the content of the questionnaire module. This will ensure that the questions included in the module are appropriate for the research needs of the study and that they are appropriate for survey respondents. We will also evaluate other modules to ensure that contain appropriate adjacent contextual variables, such as the demographic variables and behavior measures, so that the meaning of the subjective well-being measures can be understood more fully.