OMB83 C Memo

May 28, 2014

M e m o r a n d u m

TO: Shelly Martinez

Office of Management and Budget

FROM: Sarah Crissey

National Center for Education Statistics

THROUGH: Kashka Kubzdela

National Center for Education Statistics

RE: 2012/14 Beginning Postsecondary Students Longitudinal Study (BPS:12/14) Responsive Design Analysis Plan and Main Sample Data Collection Updates Change Request (OMB# 1850-0631 v.9)

Based on discussions with OMB during meetings on 4/10/2014 and 5/19/2014, NCES requests permission to amend the previously approved clearance package (OMB# 1850-0631) so as to:

1. Document the BPS:12/14 calibration sample intervention results and main sample incentive amounts, and

2. Update the BPS:12/14 responsive design analysis plan.

We have updated Supplemental Statement Parts A and B from the previously approved clearance package (OMB# 1850-0631 v.8) to reflect these two changes. This memo provides an expanded description of our plans. There is no change to the estimated respondent burden or cost to the federal government for conducting this study associated with this request.

The following paragraph was inserted at the end of section 9 in Part A:

“May 22, 2014 Addendum: Updated text in Supplemental Statement Part B (“BPS 2012-14 Full Scale Part B”) section 4b and the attached “BPS 2012-14 Full Scale Incentive Update Change Memo” summarizes the results of the incentive calibration study and the reasons for deciding to offer a $30 incentive to the remaining 90 percent of the full-scale sample in the initial Phase 1, and a $45 incentive to approximately 30 percent of the sample nonrespondents remaining at the start of Phase 3, based on the importance score model.”

Part B Edits

Amendment 1 – BPS:12/14 Calibration Sample Intervention Results and Main Sample Incentive Amounts

Based on the results of the BPS:12/14 calibration sample analyses, we amended the BPS:12/14 Full Scale clearance package (OMB# 1850-0631 v.8) to document the results of the calibration sample experiments and the final main sample incentive amounts. The following text was added in Part B, after the section entitled “Analysis of the BPS:12/14 Full Scale Responsive Design Effort”.

Calibration Study Results and Full-Scale Implementation

After approximately six weeks of data collection, response rates were compared across the 11 initial incentive levels for the calibration sample. Analyses were also conducted within propensity groups, using 5 response propensity categories. For the three highest propensity groups combined, response rates were significantly higher with each additional $5 incentive from $0 to $30, while no amount above $30 resulted in a significantly higher rate. For the two lowest response propensity groups, the only statistically significant increase in response rate was found at the $45 level. However, this finding was based on a small number of cases and was not supported by a linear trend in incentive amounts. NCES and OMB met to discuss these findings, and $30 was implemented as the initial Phase 1 promised incentive for the remaining 90 percent of the full-scale sample.

The Phase 3 responsive design experiment tested three different additional incentive amounts on the calibration sample response rates and absolute nonresponse bias estimates. After approximately 7 weeks of data collection, cases were selected for targeting based on the importance score model. Nonrespondent cases that fell below the median on the bias likelihood score or in either the highest 5 percent or lowest 15 percent of response propensities were not considered for additional targeting. From the pool of approximately 900 calibration sample nonrespondents eligible for targeting, the cases with the 500 highest importance scores (21 percent of the remaining calibration sample nonrespondents) were selected for the experiment. Cases were randomly assigned to three additional incentive treatment groups: $0, $25, or $45. Analyses showed that $45 was associated with significantly higher response rates, as well as a reduction in absolute bias across several key indicators. After consultation with OMB, NCES will use the importance model to select approximately 30 percent of the remaining sample nonrespondents at the start of Phase 3 and offer an additional $45 incentive to those targeted cases.

Amendment 2 – Updated Responsive Design Analysis Plan

We revised the text under the heading “Analysis of the BPS:12/14 Full Scale Responsive Design Effort”, on pages 17 - 19 of Part B, to include an updated analysis plan:

Analysis of the BPS:12/14 Full Scale Responsive Design Effort. Our analysis plan is based upon three premises: (1) sample cases that would contribute to nonresponse bias and have the potential to be converted to respondents can be identified at the beginning of the third and subsequent data collection phases, (2) the interventions during the third through fifth data collection phases are effective at increasing participation, and (3) increasing response rates among the targeted cases will reduce nonresponse bias by converting cases that would otherwise induce bias if they failed to respond.  In an effort to maximize the utility of this research, the analysis of the responsive design and its implementation will be described in a technical report that includes the three topics and related hypotheses described below. We intend to examine these three aspects of the BPS:12/14 responsive design and its implementation as follows:

1. Evaluate model used to target cases for under-representativeness and potential impact

To maximize the effectiveness of the BPS:12/14 responsive design approach, targeted cases need to be associated with survey responses that are underrepresented among the respondents, and the targeted groups need to be large enough to change observed estimates. In addition to assessing model fit metrics and the effective identification of cases contributing to nonresponse bias for each of the models used in the importance score calculation, the distributions of the targeted cases will be reviewed for key variables overall and within sector prior to identifying final targeted cases. During data collection, these reviews will help ensure that the cases most likely to decrease bias are targeted and that project resources are used efficiently. After data collection, similar summaries will be used to describe the composition of the targeted cases along dimensions of interest.

The importance score used to select targeted cases was calculated based on both the non-response bias potential, but also an a priori response propensity score. To evaluate how well the response propensity measure predicted actual response, we will compare the predicted response rates to observed response rates at the conclusion of data collection.

2. Evaluate the effectiveness of each phase of data collection in increasing participation.

The second key component of the BPS:12/14 responsive design is the effectiveness of the changes in survey protocol for increasing participation. Each phase introduces a feature – additional promised incentives, special field work, and an abbreviated mobile instrument. A calibration study will be used to determine the optimal baseline and Phase 3 intervention incentives.

Evaluation of the calibration study will occur during data collection so that findings can be implemented in the main subsample data collection. Approximately four weeks after the start of data collection for the calibration subsample, a logistic regression model will be estimated within each propensity decile. After estimating predicted response propensities, the minimum dollar amount associated with a predicted propensity that is not significantly different than the highest predicted propensity will be used as the baseline incentive for that propensity decile in the main subsample. Approximately four weeks after the start of the calibration study, a similar analysis will be conducted to determine the optimal incentive increase for the first phase of targeted nonrespondents.

In addition to analyzing data for the calibration study, we will evaluate response rates during the course of data collection, expecting an increase for targeted cases over the course of phases that include effective interventions. Furthermore, the cases that are not targeted with additional interventions will serve as a baseline for the pattern of responding over the course of the survey. Targeted nonrespondent completion rates are expected to increase with each phase, relative to the group that receives only standard recruitment efforts. 

3. Evaluate the ability to reduce nonresponse bias.

The rich frame, administrative, and prior wave data used in determining which cases to target for nonresponse bias reduction can, in turn, be used to evaluate (1) nonresponse bias in the final estimates and (2) changes in nonresponse bias over the course of data collection. Unweighted and weighted (using design weights) estimates of absolute relative nonresponse bias will be computed for each variable used in the models:

where  is the respondent mean and  is the full sample mean. The mean of these bias estimates can be tracked during the course of the survey and particular attention will be devoted to changes in the mean bias over each phase of data.

BPS:12/14 Responsive Design Research Questions. With the assumption that increasing the rate of response among targeted cases will reduce nonresponse bias, the BPS:12/14 responsive design experiment will explore the following research questions which build to our ultimate goal:

1. Did the a priori response propensity model predict BPS:12/14 response?

2. Are targeted respondents different from non-targeted respondents on key variables?

3. Do higher baseline incentive amounts increase early response rates?

4. Do higher additional incentive amounts increase response rates for targeted cases?

5. Did targeted cases respond at higher rates than non-targeted cases?

6. Did conversion of targeted cases reduce nonresponse bias?

7. Did conversion of high response propensity cases reduce nonresponse bias?

8. Did conversion of high bias likelihood cases reduce nonresponse bias?

Each of these questions may be stated in terms of a null hypothesis as follows:

Research question 1: Did the a priori response propensity model predict BPS:12/14 response?

H₀: At the end of data collection, there will be no association between a priori propensity predictions and observed response rates.

Research question 2: Are targeted respondents different from non-targeted respondents on key variables?

H₀: At the end of Phases 3, 4, and 5, and the end of data collection, there will be no difference between targeted respondents and non-targeted and never-targeted respondents in weighted or unweighted estimates of key variables not included in the importance score calculation.

Research question 3: Do higher baseline incentive amounts increase early response rates?

H₀: At the end of the calibration sample Phase 1, there will be no difference in response rates by baseline incentive amount.

Research question 4: Do higher additional incentive amounts increase response rates for targeted cases?

H₀: At the end of the calibration sample Phases 3, 4, and 5, there will be no difference in response rates between targeted cases that receive additional incentive offers.

Research question 5: Did targeted cases respond at higher rates than non-targeted cases?

H₀: At the end of Phases 3, 4, and 5, and at the end of data collection, there will be no difference in weighted or unweighted response rates between targeted cases and non-targeted and never-targeted cases.

Research question 6: Did conversion of targeted cases reduce nonresponse bias?

H₀: At the end of Phases 3, 4, and 5, and at the end of data collection, there will be no difference in absolute nonresponse bias when targeted cases are included in analyses compared to when they are excluded.

Research question 7:  Did conversion of high response propensity cases reduce nonresponse bias?

H₀: At the end of Phases 3, 4, and 5, and at the end of data collection, there will be no difference in absolute nonresponse bias when targeted high response propensity cases are included in analyses compared to when they are excluded.

Research question 8:  Did conversion of high bias likelihood cases reduce nonresponse bias?

H₀: At the end of Phases 3, 4, and 5, and at the end of data collection, there will be no difference in absolute nonresponse bias when targeted high bias likelihood cases are included in analyses compared to when they are excluded.