Responses to OMB comments

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: February 20, 2013

TO: Shelly Wilkie Martinez

Office of Statistical and Science Policy, Office of Management and Budget

THROUGH: Kashka Kubzdela

Office of the Deputy Commissioner, NCES

FROM: Sarah Crissey

Postsecondary, Adult, and Career Education Division, NCES

SUBJECT: Responses to the OMB Passback for BPS:12/14 Field Test (OMB # 1850-0631 v.7)

January 31, 2013 - OMB Passback

On January 31, 2013 OMB posed two questions with regard to the BPS:12/14 Supporting Statement Part B for the field test student data collection. The questions (in bold) are below:

1. “We appreciate the summary of responsive design work to date in SS Part B.  Please provide a little more explanation of how the responsive design elements of the survey build on (rather than simply repeat for a different population) work that NCES did in preparation for B&B last year.  Please also describe current thinking for how the BPS field test and the full scale first follow up fit into a larger responsive design research agenda or if not yet determined, how NCES plans to develop such an agenda.”

NCES Response: Before turning to your specific question, it is worth noting that this question highlights two important issues about not only our recent efforts to innovate around bias reduction through responsive design, but about “cross-study learning” more generally:

1. the timing of studies is often such that methodological decisions must be made for an one survey before data from another survey’s experimentation can truly inform project officers’ thinking; and

2. innovation across program areas requires sustained effort that must be intentionally managed.

While the former issue is not particularly tractable, the latter issue can—and will—be addressed by NCES’s upcoming reorganization around functional area, not student population. In his role as Senior Technical Advisor, Matthew Soldner will have responsibility for helping to ensure that methodological issues common to multiple sample surveys are addressed in such a way as to advance a consistent research agenda. While this does take place informally now, including ad hoc meetings about responsive design with ELS/HSLS program staff and colleagues at NSF, this upcoming realignment of staff responsibilities will formalize the process and, hopefully, yield visible benefits. NCES also plans to share our responsive design work more broadly, including co-organizing a session for the 2013 FedCASIC meeting with a presentation by the ELS and B&B Project Officers on the use of Mahalanobis distance measures in NCES longitudinal surveys.

That said, there are two important elements of the operational and survey methods to be used in BPS that benefit from our experience with ELS and B&B. Specifically:

1. In both ELS and B&B, the variables used in model development and distance measure calculations were largely informed by the subject matter expertise of the project officers and contractor staff. While this approach is valid on face, we propose to use information gleaned from base year non-response bias analyses to inform variable selection and operationalization for the BPS distance model. Although we would prefer to simultaneously test two models—one theoretically informed and one empirically derived—sample size in the field test environment precludes doing so. However, should this new method prove technically feasible, a head-to-head comparison may be possible in the production environment; and

2. Because responsive design in both ELS and B&B has taken place in the context of the production environment, project staff have been able to implement multiple “tools” for encouraging participation of respondents. While this can provide important information about the differential efficacy of incentives, increased effort, or other interventions, we have come to realize that their use makes it more difficult to isolate the treatment effect of differentially targeting cases via a distance measure. In BPS, we recommend the use of a consistent intervention at each time point, thereby more readily answering the question of whether it was the specific timing of an intervention or the use of targeting via Mahalanobis that was related to the anticipated reduction in non-response bias. (We are keenly aware that OMB supports statistical agencies to continue to innovate around the notion of a more robust “toolbox.” We concur entirely, and believe that, given issues related to statistical power, such work is best done in the production environment.)

2. In SS Part A -- From our subject matter experts on financial aid policy:

Page 5:  “Tuition costs, financial aid, and student debt are also receiving considerable attention by lawmakers. The continued rise of tuition costs has prompted the Obama Administration to propose shifting campus-based aid to institutions that keep costs down and to propose a Race to the Top for postsecondary education, with the goal of improving college affordability and completion. The Student Loan and Fiscal Responsibility Act of 2009 (SAFRA) temporarily lowered interest rates on student loans and increased Pell Grant amounts while restricting Pell Grant eligibility to shore up costs. Faced with government deficits, the extent to which these current rules will be continued is uncertain. However, student debt is increasingly seen as a hardship and potential barrier for college graduates, and BPS can provide insight into the extent to which college costs, financial aid, and debt are related to persistence, attainment, and employment outcomes.”

There are several errors in this paragraph. 

• “propose shifting aid to institutions that keep costs down” is limited to some student aid programs and that sentence makes it seem broad based.   I added “campus based” to that sentence.

• SAFRA did not lower interest rates and did not restrict eligibility for Pell Grants (that was done in subsequent pieces of legislation)”.  I propose to delete that sentence entirely. 

• I would also take out the sentence in red.

NCES Response: We appreciate the correction and have updated Part A accordingly.

February 12, 2013 - OMB Passback

On February 12, 2013 OMB posed three additional questions with regard to the BPS:12/14 Supporting Statement Part B for the field test student data collection. The questions (in bold) are below:

On page 21 (Table 5), we don’t understand how there are 609 high distance cases, only 294 of whom receive the treatment if all 609 are in the experimental group.  Aren’t all high distance cases in the experimental group getting the higher incentive?  Please explain.

NCES Response: In the spreadsheet which produced the original table 5, a 5 percent rate, rather than a 30 percent rate, was used for identifying the high distance cases for the first Mahalanobis distance calculation. Hence, the actual number of high distance cases should have been higher. The numbers have been corrected, as shown in table 5 below and in the revised Part B.

We are anticipating that, across the control and treatment groups, there will be approximately 682 high distance cases identified, half of which will be in the control group and half in the treatment group. All high distance cases in the experimental group will receive the treatment of the higher incentive.

We have tried to clarify the relationships between the hypotheses being tested and table 5. Specifically, Hypothesis 1 is measuring the effect of a treatment. Hypotheses 2a and 2b are just identifying the groups being compared—the comparison group (high distance) includes both treated and nontreated cases. In table 5, we changed the column heading to reflect that the “Experimental group” column also includes comparison groups as well.

Also, what is the basis for estimating such different response rates between the control and experimental groups (47% versus 17% by our calculations)?

NCES Response: The original table 5 listed 792 normal distance respondents out of 1,692 normal distance cases (46.8 percent) and 101 high distance respondents of the 609 high distance cases (16.6 percent). Those calculations resulted from error in initial selection rates in the spreadsheet. In the revised table 5, shown above and in part B, the response rates are now 51.4 percent and 51.3 percent for normal and high distance groups, respectively. The expectation is that approximately 41 percent of completed interviews will have already occurred during the first 5 weeks of data collection, prior to the first Mahalanobis calculation.

Finally, we would consider changing the alpha to .10 in statistical testing given the developmental nature of this test and the relatively low power.”

NCES Response: Table 5, shown below, has been revised for both an alpha of 0.05 and of 0.10.

Table 5. Detectable differences for experimental hypotheses

Hypothesis	Control group			Experimental/Comparison group			Detectable difference at alpha
	Definition	Sample size		Definition	Sample size		 = .05	 = .10
1	High-distance cases that did not receive treatment	341		High distance cases that ever received treatment ($55)	341		9.3	7.8
2a	Normal distance cases	1,154		High distance cases	682		8.1	6.9
2b	Normal distance respondents	593		High distance respondents	350		11.7	1.0
2c	All respondents, excluding high-distance experimental cases,	2,405		All respondents, excluding high-distance control cases	2,446		4.5	3.8
3	Eligible cases, excluding high-distance treatment cases	2,616		Eligible cases, excluding high-distance control cases	2,616		2.5	2.2