Sample Design

Appendix F
Sample Designs for the BPS:04/06
and BPS:04/09 Full-scale Studies

Appendix F
Sample Designs for the BPS:04/06
and BPS:04/09 Full-scale Studies

The sample design for the base-year study (NPSAS:04), in which the BPS:04/06 and BPS:04/09 cohort was identified, was described in appendix E. This appendix provides a description of the sample design for the second follow-up of the BPS:04 cohort (BPS:04/09). The students in the BPS;04/09 will be those who were determined to be BPS-eligible after the BPS:04/06 data collection. For this reason, the BPS:04/06 sample design and results are discussed first, followed by the BPS:04/09 sample design.

F.1 BPS:04/06 Student Sample

Students eligible for the BPS:04/06 full-scale study were those both eligible to participate in NPSAS:04 and identified as FTB students at NPSAS sample institutions in the 2003–04 academic year. Consistent with previous NPSAS studies, the students eligible for the NPSAS:04 full-scale study were those enrolled in eligible institutions who satisfied all the following eligibility requirements:

• were enrolled in either (1) an academic program; (2) at least one course for credit that could be applied toward fulfilling the requirements for an academic degree; or (3) an occupational or vocational program that required at least 3 months or 300 clock hours of instruction to receive a degree, certificate, or other formal award; and

• were not concurrently or solely enrolled in high school, or in a General Educational Development (GED) or other high school completion program.

NPSAS-eligible students who enrolled in a postsecondary institution during the “NPSAS year” (July 1, 2003–June 30, 2004) for the first time after completing high school were considered pure FTBs and were eligible for BPS:04/06. Those NPSAS-eligible students who had enrolled for at least one course after completing high school but had never completed a postsecondary course before the 2003–04 academic year were considered effective FTBs and were also eligible for the BPS:04/06 full-scale study. In the full-scale BPS data collection, we sampled from both (1) NPSAS:04 respondents who were identified as (pure or effective) FTBs and (2) NPSAS:04 nonrespondents who were potential (pure or effective) FTBs.

The BPS:04/06 student sample consisted of four groups according to their base-year response status:

1. NPSAS:04 study respondents who completed the student interview and were determined to be FTBs;

2. NPSAS:04 study respondents who completed the student interview but were initially determined to be non-FTB other undergraduates, and who were potential FTBs based on data from other sources;

3. a subsample of potential FTBs¹ who were NPSAS:04 study respondents but student interview nonrespondents; and

4. a subsample of potential FTBs who were NPSAS:04 study nonrespondents.

Multiple data sources were used to provide information regarding a student’s FTB status during the NPSAS year, including the NPSAS:04 student interview, records from the student’s base-year institution via CADE, and federal financial aid sources. The data elements that were examined to estimate a student’s likelihood of being an FTB and to construct the frame for the BPS:04/06 sample included the following:

• indicator of FTB status from the institution enrollment lists used for NPSAS:04 student sampling;

• indicator of FTB status from the Central Processing System (CPS);

• indicator of FTB status from student-level data obtained from institutional records via CADE;

• student reports (obtained during the NPSAS:04 interview) indicating that they were FTBs during the 2003–04 academic year;

• year of high school graduation;

• receipt of Stafford loan (date loan was first received and number of years loan was received);

• receipt of Pell grant (date grant was first received and number of years grant was received); and

• undergraduate class level.

Using the above indicators, a set of decision rules was developed to identify which cases would be included or excluded from the follow-up sample, and which among those included would require additional eligibility screening.

The NPSAS:04 sample yielded the numbers of students below who either indicated that they were FTBs during the interview and had other institutional records or federal financial aid sources that supported this, or were identified as potential FTBs based on institutional records or federal financial aid sources:

• 24,994 students responding to the student interview indicated that they were FTBs during the 2003–04 academic year. Based on a review of the FTB status indicators above, 21,170 of these were identified for inclusion in the follow-up sample. Of the 21,170 included in the follow-up sample, 19,798 had other data that strongly supported their FTB status, and 1,372 of these students had some indications that they were not FTBs; these potential “false positives” were rescreened during the BPS:04/06 interview to confirm their status. The remaining 3,824 of the original 24,994 were identified for exclusion from the follow-up when multiple data sources confirmed that they could not have been FTBs during the NPSAS year.

• 1,423 students were not originally classified as FTBs, but were potential FTBs based on either CPS data or because they had a high school graduation date in 2003 or 2004; these potential “false negatives” were also screened during the BPS:04/06 interview to verify their status.

• 8,863 students did not respond to the student interview but were classified as NPSAS:04 study respondents and were potential FTBs based on CADE or CPS data, more positive than negative indicators among the other variables, and any Stafford loans or Pell grants that began after 2003.

• 719 NPSAS:04 sample members were potential FTBs based on information from CADE or CPS, but did not respond to the student interview and did not have sufficient data to be classified as study respondents.

Table F-1 summarizes the distribution of the BPS:04/06 sample.

Table F-1. Distribution of BPS:04/06 full-scale sample, by base-year response status

Base-year response status	Number of cases
Group 1
Base-year study respondent student interview respondents who were classified as FTBs1	24,994
Total to be included in sample	21,170
No additional screening required	19,798
Additional screening required	1,372
Group 2
Base-year study respondent student interview respondents who were classified as Other Undergraduate (potential false negatives)	28,605
Likely FTB	1,423
Group 3
Potential FTB: base-year study respondent student interview nonrespondents	10,172
Likely FTB	8,863
Subsample	460
Group 4
Potential FTB: base-year study nonrespondents	3,887
Likely FTB	719
Subsample	40
Final sample	23,093

¹ Due to evidence indicating they were not eligible for inclusion in the cohort of FTBs, 3,824 base-year study respondents were removed from the follow-up sample.

NOTE: FTB = first-time beginner.

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary Student Aid Study (NPSAS:04).

As noted earlier, 9,582 NPSAS:04 student interview nonrespondents were classified as potential FTBs. Of these, 8,863 were NPSAS:04 study respondents who did not respond to the student interview, and 719 were NPSAS:04 study nonrespondents. NPSAS:04 student interview nonrespondents who were potential FTBs were subsampled for follow-up to improve the nonresponse bias reduction achieved through the nonresponse adjustments incorporated into the NPSAS:04 statistical analysis weights. For these students, sampling types were developed from the following characteristics:

• likelihood of being an FTB (medium, high); and

• tracing outcome (located, not located).²

Two factors, stratification by tracing outcome and the likelihood of being an FTB, were used to oversample the students most likely to be located and eligible for the study. The frame was also sorted by institutional sector to ensure representativeness of the sample.

A stratified sample of 500 was selected with probabilities proportional to their NPSAS:04 sampling weights. Table F-2 summarizes the BPS:04/06 counts of students eligible for the sample and the sample sizes, including the allocation of the subsample of 500 cases to the two groups of NPSAS:04 student interview nonrespondents. Given that the NPSAS:04 sampling weights were available for all student interview nonrespondents, they served as the basis for computing the BPS:04/06 analysis weights. Therefore, selection of the NPSAS:04 student interview nonrespondents with probabilities proportional to these weights was used to reduce the overall unequal weighting effects for the sample.

The BPS:04/06 sample consisted of 23,093 students. At the conclusion of the BPS:04/06 data collection, 17,707 students were initially determined to be eligible respondents, 4,548 were nonrespondents, and 838 were ineligible. Logistic models were developed to predict which of the nonrespondents were eligible. As a result of this step, 4,482 nonrespondents were classified as eligible. Additional cases were determined to be ineligible after comparing the BPS:04/06 sample members with the National Student Clearinghouse data, resulting in 18,644 eligible sample members.

Table F-3 shows the numbers of students determined to be eligible for BPS:04/06 through screening, modeling, and the comparison to the National Student Clearinghouse data, and also the number of respondents to the BPS:04/06 interview by type of institution.

All of the 18,644 students who were determined to be eligible for BPS:04/06 were retained on the analytic data file for BPS:04/06. Almost all of these students had some data from the NPSAS:04 interview or other data sources; BPS:04/06 interview data was imputed for the nonrespondents using a weighted hot deck imputation procedure.

Weights were constructed for use in analyzing the BPS:04/06 data. Starting with the NPSAS:04 weight, an adjustment was applied for subsampling the NPSAS interview nonrespondents. These weights were trimmed and smoothed, and calibrated to the total numbers of students by institution type who were enrolled in the Fall of 2003, and those who were enrolled full time; the calibration totals were obtained from IPEDS.

Table F-2. BPS:04/06 sample allocation for NPSAS:04 student interview, by type of student: 2006

Type of student	Students eligible for sample	Sample size
Total	32,175	23,093
NPSAS:04 student interview respondents classified as FTBs during interview	21,170	21,170
Likely to be FTBs	19,798	19,798
Potential to be FTBs	1,372	1,372
NPSAS:04 student interview respondents who were potentially FTBs but were not classified as FTBs during interview	1,423	1,423
NPSAS:04 student interview nonrespondents	9,582	500
Study respondents who were student interview nonrespondents	8,863	460
Located, likely to be FTBs	3,591	275
Located, potential to be FTBs	552	30
Not located	4,720	155
Study nonrespondents	719	40
Located, likely to be FTBs	88	12
Located, potential to be FTBs	246	12
Not located	385	16

NOTE: The likelihood of being an FTB was determined from student financial aid data and institutional record (CADE) data and based on the number and type of indicators suggesting a student was an FTB. The location information was based on whether the advance tracing information from the 2004/06 Beginning Postsecondary Students Longitudinal Study (BPS:04/06) either confirmed the existing telephone number or yielded a new telephone number. Eligibility rates were assumed to be lower for NPSAS:04 study nonrespondents since less information was available for these students. FTB = first-time beginner.

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2003–04 National Postsecondary Student Aid Study (NPSAS:04) and 2004/06 Beginning Postsecondary Students Longitudinal Study (BPS:04/06).

Table F-3. BPS:04/06 eligible students and interview respondents, by institution type: 2006

Type of institution	Eligible students
	Total	Number of respondents
Total	18,644	14,901
Public, less-than-2-year	547	424
Public, 2-year	6,346	4,834
Public, 4-year non-doctorate-granting	1,628	1,348
Public, 4-year doctorate-granting	3,153	2,724
Private not-for-profit less-than-4-year	533	403
Private not-for-profit 4-year non-doctorate-granting	2,130	1,867
Private, not-for-profit, 4-year doctorate-granting	1,688	1,481
Private for-profit, less-than-2-year	1,461	1,004
Private for-profit 2-year-or-more	1,158	816

F.2 BPS:04/09 Student Sample

The BPS:04/09 sample will consist of all of the 18,644 student who were determined to be eligible during BPS:04/06. Table F-4 shows the sample distribution by prior response status (i.e., whether the student responded to the NPSAS:04 CATI interview and the BPS:04/06 interview). This table also shows that BPS:04/09 will have an expected 15,740 respondents. The estimated response rate for the BPS:04/09 full-scale study was estimated using the response rates obtained in the BPS:96/01 field test and full-scale studies.

Consistent with the BPS:04/06 data collection, all of the sample members (regardless of response status) will be included on the BPS:04/09 data file. Imputation methods, such as weighted hot deck, will be used to fill in missing item data for interview respondents as well as nonrespondents.

Table F4. BPS:04/09 Full-scale study sample size and expected number of respondents, by response status to NPSAS:04 and BPS:04/06

NPSAS:04 study respondent	NPSAS:04 CATI respondent	BPS:04/06 respondent	Number of cases	Percent expected response rate	Expected number of respondents
Total			18,644	84	15,740
Yes	Yes	Yes	14,751	89	13,128
Yes	Yes	No	3,510	66	2,317
Yes	No	Yes	138	83	115
Yes	No	No	218	75	164
No	No	Yes	12	80	10
No	No	No	15	50	8

1 A “potential FTB” is one who is expected to have been a first-time beginning student during the “NPSAS year” (July 1, 2003–June 30, 2004) but was not confirmed as such during the student interview. Students were identified as potential FTBs by their sample institution. Other data sources (CPS, CADE) also provide an indication of FTB status for the time period of interest.

2 The results from the advance tracing for BPS:04/06 were used to determine whether a student had been located. The National Change of Address file (NCOA) was used to obtain updated addresses for the student, then Telematch was used to obtain an updated telephone number. The student was classified as “located” if Telematch either returned a new telephone number or confirmed the current telephone number.

BPS:04/09 Supporting Statement Request for OMB Review (SF83i) F-0