BPS SS B revised without track changes 12 08.rtf

2. Collection of Information Employing Statistical Methods

This submission requests clearance for the BPS:04/09 full-scale study to be conducted in 2009. A separate submission requested clearance for the BPS:04/09 full-scale transcript collection. The sampling plan for the full-scale study is presented below and describes the base-year and first follow-up samples as well as the proposed sample for the second follow-up. Also discussed is the process used to determine BPS eligibility in the full-scale administrations of the base year and first follow-up studies to inform sampling and eligibility determination for the BPS:04/09 full-scale study.

1. Respondent Universe

The respondent universe for the full-scale data collection consists of beginning (FTBs) students enrolled for the first time during the 2003–04 academic year. The institution and student universes are described in greater detail in the subsections that follow.

The institutions eligible for NPSAS:04 were required during the 2003–04 academic year to:

• offer an educational program designed for persons who have completed secondary education;

• offer at least one academic, occupational, or vocational program of study lasting at least 3 months or 300 clock hours;

• offer courses that are open to more than the employees or members of the company or group (e.g., union) that administers the institution;

• be eligible to distribute Title IV aid;

• be located in the 50 states, the District of Columbia, or Puerto Rico; and

• be other than a U.S. Service Academy.

Institutions providing only vocational, recreational, or remedial courses or only in-house courses for their own employees were excluded. U.S. Service Academies were excluded because of their unique funding/tuition base.

The above institutional eligibility conditions are completely consistent with previous NPSAS studies with two exceptions. The requirement to be eligible to distribute Title IV aid was implemented beginning with NPSAS:2000.¹ Also, the previous NPSAS studies excluded institutions that only offered correspondence courses. NPSAS:04 includes such institutions if they are eligible to distribute Title IV student aid.

The institutional sampling frame for NPSAS:04 was constructed from the 2001 Integrated Postsecondary Education Data System (IPEDS) Institutional Characteristics (IC) file, the 2001 IPEDS Completions file, and the 2001 IPEDS Fall Enrollment file. The sample for NPSAS:04 was selected prior to selection of the field test institutions. Then, the sample of field test institutions was selected purposively from the complement of the full-scale sample institutions. This ensured that no institutions were in both the field test and full-scale samples without affecting the representativeness of the full-scale sample.

Records on the IPEDS IC file that did not represent NPSAS-eligible institutions were deleted. Hence, records that represented central offices, U.S. service academies, or institutions located outside the United States and Puerto Rico were deleted. The IPEDS files were then “cleaned” to resolve the following types of problems:

• missing or zero enrollment or completions data, because these data are needed to compute measures of size for sample selection; and

• unusually large or small enrollment, especially if imputed, because, if incorrect, these data would result in inappropriate probabilities of selection and sample allocation.

Table 7 presents the allocation of the NPSAS:04 institutional sample to the nine institutional sampling strata. The number of sample institutions is 1,500, accounting for historical rates of participation in CADE, institution eligibility rates, and rates with which sample institutions provide student lists for sample selection. Table 7 shows the resulting institutional sample sizes, which was 1,370 institutions providing lists for sample selection and 1,285 institutions providing CADE data.

We selected a direct, unclustered sample of institutions, like the sample selected for NPSAS:2000 and NPSAS:96, rather than a clustered sample like those used for previous NPSAS studies. A subset of approximately 1,000 institutions selected for NPSAS was also in the 2004 National Study of Postsecondary Faculty (NSOPF:04) sample. In addition, to allow analysis of the effects of state tuition and student aid policies in individual states, representative samples of institutions were selected from three strata—public 2-year institutions; public 4-year institutions; and private not-for-profit 4-year institutions—in each of the following 12 states: CA, CT, DE, GA, IL, IN, MN, NE, NY, OR, TN, and TX.

Table 7. NPSAS:04 institution sample sizes and yield, by sampling strata

Institutional sector	Institutions
	Frame	Sample	Eligible	List respondents	CADE respondents
Total	6,674	1,500	1,483	1,370	1,285
Public less-than-2-year	321	50	48	41	37
Public 2-year	1,225	322	319	303	288
Public 4-year non-doctorate-granting	358	150	150	143	136
Public 4-year doctorate granting	276	251	251	238	226
Private not-for-profit 2-year or less	379	60	55	52	48
Private not-for-profit, 4-year non-doctorate granting	1,076	252	249	212	195
Private not-for-profit 4-year doctorate granting	537	165	165	155	147
Private for-profit less-than-2-year	1,390	150	146	131	118
Private for-profit 2-year or more	1,112	100	100	95	90

NOTE: Institution counts based on the Fall 2000 IPEDS data collection. Institution eligibility rate: 98.9 percent. Institution list response rate: 92.4 percent. 1,000 of the 1,500 institutions also are in the NSOPF:2004 sample.

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary Student Aid Study, “Field Test Methodology Report” (NPSAS:04).

Students eligible for the BPS:04/06 full-scale study were those both eligible to participate in NPSAS:04 and identified as FTB students at NPSAS sample institutions in the 2003–04 academic year. Consistent with previous NPSAS studies, the students eligible for the NPSAS:04 full-scale study were those enrolled in eligible institutions who satisfied all the following eligibility requirements:

• were enrolled in either (1) an academic program; (2) at least one course for credit that could be applied toward fulfilling the requirements for an academic degree; or (3) an occupational or vocational program that required at least 3 months or 300 clock hours of instruction to receive a degree, certificate, or other formal award; and

• were not concurrently or solely enrolled in high school, or in a General Educational Development (GED) or other high school completion program.

NPSAS-eligible students who enrolled in a postsecondary institution during the “NPSAS year” (July 1, 2003–June 30, 2004) for the first time after completing high school requirements were considered pure FTBs and were eligible for BPS:04/06. Those NPSAS-eligible students who had enrolled for at least one course after completing high school but had never completed a postsecondary course before the 2003–04 academic year were considered effective FTBs and were also eligible for the BPS:04/06 full-scale study. In the full-scale BPS data collection, students were sampled from both (1) NPSAS:04 respondents who were identified as (pure or effective) FTBs and (2) NPSAS:04 nonrespondents who were potential (pure or effective) FTBs.

2. Statistical Methodology

This section provides description of the sample design for the second follow-up with the BPS:04 cohort (BPS:04/09). The students included in the BPS:04/09 sample will be all of those who were determined to be BPS-eligible after the BPS:04/06 data collection. For this reason, the BPS:04/06 sample design and results are discussed first, followed by the BPS:04/09 sample design. The sample design for the base-year study (NPSAS:04), from which the BPS:04 cohort was identified, was described in the NPSAS:04 Full Scale Methodology Report.²

The BPS:04/06 student sample consisted of four groups according to their base-year response status:

1. NPSAS:04 study respondents who completed the student interview and who were determined to be FTBs;

2. NPSAS:04 study respondents who completed the student interview but who were initially determined to be non-FTB other undergraduates, and who were potential FTBs based on data from other sources;

3. a subsample of potential FTBs³ who were NPSAS:04 study respondents but student interview nonrespondents; and

4. a subsample of potential FTBs who were NPSAS:04 study nonrespondents.

Multiple data sources were used to provide information regarding a student’s FTB status during the NPSAS year, including the NPSAS:04 student interview, records from the student’s base-year institution via CADE, and federal financial aid sources. The data elements that were examined to estimate a student’s likelihood of being an FTB and to construct the frame for the BPS:04/06 sample included the following:

• indicator of FTB status from the institution enrollment lists used for NPSAS:04 student sampling;

• indicator of FTB status from the Central Processing System (CPS);

• indicator of FTB status from student-level data obtained from institutional records via CADE;

• student reports (obtained during the NPSAS:04 interview) indicating that they were FTBs during the 2003–04 academic year;

• year of high school graduation;

• receipt of Stafford loan (date loan was first received and number of years loan was received);

• receipt of Pell grant (date grant was first received and number of years grant was received); and

• undergraduate class level.

Using the above indicators, a set of decision rules was developed to identify which cases would be included or excluded from the follow-up sample, and which among those included would require additional eligibility screening.

The NPSAS:04 sample yielded the numbers of students below who either indicated that they were FTBs during the interview and had other institutional records or federal financial aid sources that supported this, or were identified as potential FTBs based on institutional records or federal financial aid sources:

• 24,994 students responding to the student interview indicated that they were FTBs during the 2003–04 academic year. Based on a review of the FTB status indicators above, 21,170 of these were identified for inclusion in the follow-up sample. Of the 21,170 included in the follow-up sample, 19,798 had other data that strongly supported their FTB status, and 1,372 of these students had some indications that they were not FTBs; these potential “false positives” were rescreened during the BPS:04/06 interview to confirm their status. The remaining 3,824 of the original 24,994 were identified for exclusion from the follow-up when multiple data sources confirmed that they could not have been FTBs during the NPSAS year.

• 1,423 students were not originally classified as FTBs, but were potential FTBs based on either CPS data or because they had a high school graduation date in 2003 or 2004; these potential “false negatives” were also screened during the BPS:04/06 interview to verify their status.

• 8,863 students did not respond to the student interview but were classified as NPSAS:04 study respondents and were potential FTBs based on CADE or CPS data, more positive than negative indicators among the other variables, and any Stafford loans or Pell grants that began after 2003.

• 719 NPSAS:04 sample members were potential FTBs based on information from CADE or CPS, but did not respond to the student interview and did not have sufficient data to be classified as study respondents.

Table 8 summarizes the distribution of the BPS:04/06 sample.

Table 8. Distribution of BPS:04/06 full-scale sample, by base-year response status

Base-year response status	Number of cases
Group 1
Base-year study respondent student interview respondents who were classified as FTBs1	24,994
Total to be included in sample	21,170
No additional screening required	19,798
Additional screening required	1,372
Group 2
Base-year study respondent student interview respondents who were classified as Other Undergraduate (potential false negatives)	28,605
Likely FTB	1,423
Group 3
Potential FTB: base-year study respondent student interview nonrespondents	10,172
Likely FTB	8,863
Subsample	460
Group 4
Potential FTB: base-year study nonrespondents	3,887
Likely FTB	719
Subsample	40
Final sample	23,093

¹ Due to evidence indicating they were not eligible for inclusion in the cohort of FTBs, 3,824 base-year study respondents were removed from the follow-up sample.

NOTE: FTB = first-time beginner.

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary Student Aid Study (NPSAS:04).

As noted earlier, 9,582 NPSAS:04 student interview nonrespondents were classified as potential FTBs. Of these, 8,863 were NPSAS:04 study respondents who did not respond to the student interview, and 719 were NPSAS:04 study nonrespondents. NPSAS:04 student interview nonrespondents who were potential FTBs were subsampled for follow-up to improve the nonresponse bias reduction achieved through the nonresponse adjustments incorporated into the NPSAS:04 statistical analysis weights. For these students, sampling types were developed from the following characteristics:

• likelihood of being an FTB (medium, high); and

• tracing outcome (located, not located).⁴

Two factors, stratification by tracing outcome and the likelihood of being an FTB, were used to oversample the students most likely to be located and eligible for the study. The frame was also sorted by institutional sector to ensure representativeness of the sample.

A stratified sample of 500 was selected with probabilities proportional to their NPSAS:04 sampling weights. Table 9 summarizes the BPS:04/06 counts of students eligible for the sample and the sample sizes, including the allocation of the subsample of 500 cases to the two groups of NPSAS:04 student interview nonrespondents. Given that the NPSAS:04 sampling weights were available for all student interview nonrespondents, they served as the basis for computing the BPS:04/06 analysis weights. Therefore, selection of the NPSAS:04 student interview nonrespondents with probabilities proportional to these weights was used to reduce the overall unequal weighting effects for the sample.

Table 9. BPS:04/06 sample allocation for NPSAS:04 student interview, by type of student: 2006

Type of student	Students eligible for sample	Sample size
Total	32,175	23,093
NPSAS:04 student interview respondents classified as FTBs during interview	21,170	21,170
Likely to be FTBs	19,798	19,798
Potential to be FTBs	1,372	1,372
NPSAS:04 student interview respondents who were potentially FTBs but were not classified as FTBs during interview	1,423	1,423
NPSAS:04 student interview nonrespondents	9,582	500
Study respondents who were student interview nonrespondents	8,863	460
Located, likely to be FTBs	3,591	275
Located, potential to be FTBs	552	30
Not located	4,720	155
Study nonrespondents	719	40
Located, likely to be FTBs	88	12
Located, potential to be FTBs	246	12
Not located	385	16

NOTE: The likelihood of being an FTB was determined from student financial aid data and institutional record (CADE) data and based on the number and type of indicators suggesting a student was an FTB. The location information was based on whether the advance tracing information from the 2004/06 Beginning Postsecondary Students Longitudinal Study (BPS:04/06) either confirmed the existing telephone number or yielded a new telephone number. Eligibility rates were assumed to be lower for NPSAS:04 study nonrespondents since less information was available for these students. FTB = first-time beginner.

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2003–04 National Postsecondary Student Aid Study (NPSAS:04) and 2004/06 Beginning Postsecondary Students Longitudinal Study (BPS:04/06).

The BPS:04/06 sample consisted of 23,093 students. At the conclusion of the BPS:04/06 data collection, 17,707 students were initially determined to be eligible respondents, 4,548 were BPS interview nonrespondents, and 838 were ineligible. Logistic models were developed to predict which of the nonrespondents were eligible. As a result of this step, 4,482 nonrespondents were classified as eligible. Additional cases were determined to be ineligible after comparing the BPS:04/06 sample members with the National Student Clearinghouse data, resulting in 18,644 eligible sample members.

Table 10 shows the numbers of students determined to be eligible for BPS:04/06 through screening, modeling, and the comparison to the National Student Clearinghouse data, and also the number of respondents to the BPS:04/06 interview by type of institution.

All of the 18,644 students who were determined to be eligible for BPS:04/06 were retained on the analytic data file for BPS:04/06. Almost all of these students had some data from the NPSAS:04 interview or other data sources; BPS:04/06 interview data were imputed for the nonrespondents using a weighted hot deck imputation procedure.

Table 10. BPS:04/06 eligible students and interview respondents, by institution type: 2006

Type of institution	Eligible students
	Total	Number of respondents
Total	18,644	14,901
Public, less-than-2-year	547	424
Public, 2-year	6,346	4,834
Public, 4-year non-doctorate-granting	1,628	1,348
Public, 4-year doctorate-granting	3,153	2,724
Private not-for-profit less-than-4-year	533	403
Private not-for-profit 4-year non-doctorate-granting	2,130	1,867
Private, not-for-profit, 4-year doctorate-granting	1,688	1,481
Private for-profit, less-than-2-year	1,461	1,004
Private for-profit 2-year-or-more	1,158	816

NOTE: The first column gives the number of students who were included in the BPS:04/06 sample and were determined to be first-time beginners (FTBs) using NPSAS:04 data, BPS:04.06 interview data, loan sources, National Student Clearinghouse data, and the CPS data; this is the number of students who were included on the BPS:04/06 data file and will be included in the BPS:04/09 data collection. The second column gives the numbers of students who responded to the BPS:04/06 interview and were determined to be FTBs.

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2003–04 National Postsecondary Student Aid Study (NPSAS:04) and 2004/06 Beginning Postsecondary Students Longitudinal Study (BPS:04/06).

Weights were constructed for use in analyzing the BPS:04/06 data. Starting with the NPSAS:04 weight, an adjustment was applied for subsampling the NPSAS interview nonrespondents. These weights were first trimmed and smoothed within the original NPSAS sampling strata so that the weights for the subsample of NPSAS:04 interview nonrespondents did not cause a large increase in the unequal weighting design effect. These weights were then calibrated to the total numbers of students by institution type who were enrolled in the fall of 2003 and those who were enrolled full time; the calibration totals were obtained from IPEDS.

The BPS:04/09 sample will consist of all of the 18,644 students who were determined to be eligible during BPS:04/06, except for any who are determined to have died since that data collection. In other words, the BPS:04/09 sample will consist of all students who were determined to be eligible for BPS:04/06 and were included on the BPS:04/06 data file (regardless of their BPS:04/06 response status). Table 11 shows the sample distribution by prior response status (i.e., whether the student responded to the NPSAS:04 interview and the BPS:04/06 interview). The table also shows that the BPS:04/09 interview will have an expected 14,916 respondents. The estimated response rate for the BPS:04/09 full-scale study was estimated using the response rates obtained in the BPS:96/01 and BPS:04/06 full-scale, and BPS:04/09 field test studies.

Consistent with the BPS:04/06 data collection, all of the sample members who are still alive (regardless of response status) will be included on the BPS:04/09 data file. The final BPS:04/06 analysis weight variable will be used for analyzing the BPS:04/09 cases. Imputation methods, such as weighted hot deck, will be used to fill in missing item data for interview respondents as well as nonrespondents.

Table 11. BPS:04/09 full-scale study sample size and expected number of respondents, by response status to NPSAS:04 and BPS:04/06

NPSAS:04 study respondent	NPSAS:04 interview respondent	BPS:04/06 respondent	Number of cases	Percent expected response rate1	Expected number of respondents
Total			18,644	80	14,916
Yes	Yes	Yes	14,751	85	12,539
Yes	Yes	No	3,510	60	2,106
Yes	No	Yes	138	75	104
Yes	No	No	218	70	153
No	No	Yes	12	60	8
No	No	No	15	40	6

NOTE: This table shows the distribution of the BPS:04/09 sample and expected number of respondents by response status to the NPSAS:04 interview and the BPS:04/06 interview.

¹The estimated response rate for the BPS:04/09 full-scale study was estimated using the response rates obtained in the BPS:96/01 field test and full-scale studies.

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2003–04 National Postsecondary Student Aid Study (NPSAS:04) and 2004/06 Beginning Postsecondary Students Longitudinal Study (BPS:04/06).

3. Methods for Maximizing Response Rates

Response rates for BPS:04/09 will be a function of success in two basic activities: tracing and locating the sample members and, once contacted, gaining their cooperation and time to complete the interview in one of three modes: a self-administered web-based interview, a telephone interview, or an in-person field interview. We will use an advance tracing procedure initially and follow this by more intensive procedures, including field interviewing and the use of incentives, as necessary. Given the tracing, self-administration, telephone interviewing, and field interviewing plans we discuss below, we are confident that we will be able to achieve a response rate of 80 percent for the BPS:04/09 full-scale study. The plans proposed for the full-scale data collection were tested during the field test collection conducted in 2008.

1. Tracing and Interviewing Sample Members

One of the main issues for the BPS:04/09 data collection effort will be locating the members of the sample cohort. While most of these individuals have not been contacted since the BPS:04/06 first follow-up interview, some will have not been contacted since the NPSAS:04 base-year interview. Members of the cohort are highly mobile, most likely to only use a cellular telephone, rather than a landline, and may have completed their degrees, transferred to a different college, or moved. The high mobility rate of this population presents challenges to the BPS:04/09 tracing effort.

A successful tracing operation is dependent on a multitude of factors including the characteristics of the population to be located, the age of the locating information for the population (approximately 2 to 5 years old at the start of data collection), and the completeness and accuracy of that information. To maximize our location rate, sufficient resources will be devoted to tracing operations both in-house and in the field, giving careful consideration to identifying and implementing the most effective, yet cost efficient, tracing strategies for this population. The locator database for the cohort includes critical tracing information for most of the sample members, including their previous residences and telephone numbers. Moreover, Social Security numbers are available for virtually all of the sample members (99 percent), as well as other information useful for tracing.

To achieve the desired response rate in the full-scale study required by NCES standards, we propose a multistage tracing approach that will capitalize on the availability of locating data obtained in the previous rounds and the continuing cooperation of sample members. This multistage approach will consist of several steps designed to yield the maximum number of locates with the least expense. The steps of our multistage tracing plan include the following elements.

Advance tracing. We propose to employ an advance tracing operation prior to full-scale data collection that will update the addresses of sample members. Advance tracing will include searches of the U.S. Department of Education’s CPS for information on financial aid recipients. We will also conduct computerized searches of other databases, including the National Change of Address (NCOA), Telematch and ComServ's Death Information System. We will compare all sample member addresses obtained from the locator database (with information from NPSAS:04 and BPS:04/06) against the NCOA and Telematch databases to identify sample members who have moved since the previous follow-up. Updated addresses and telephone numbers produced by these advance tracing activities will be entered into the BPS:04/09 locator database and made available to data collection personnel at the start of data collection.

Advance interactive tracing. After the completion of advance tracing, cases without good locating information (primarily cases that participated only in NPSAS:04) will be directed for additional interactive tracing. Specially trained tracing staff will perform intensive tracing to locate additional contact information for these cases. In many instances, this will involve an interactive credit bureau search and may involve other interactive databases of locator information.

Parent mailing. We will mail a letter to the parents of all sample members informing them that their child’s participation will be requested. This letter will also include a study brochure, address update form, and a business reply envelope.

Initial contact mailing to sample members. We will mail a personalized letter (signed by the acting NCES Commissioner), study brochure, address update form, and business reply envelope to all sample members. This letter will include the study's website address and toll-free telephone number, and will request that sample members update their postal and electronic mail addresses. Undeliverable mailings to sample members will be recorded, and the next best address will be used to resend the materials. Once all potential addresses for the sample member are exhausted, we will contact other information sources for the sample member (e.g., a parent, other relative, or a designated contact.)

Data collection announcement mailing to sample members. Once we have the most current contact information for the sample members, we will mail a postcard to the sample cohort announcing the start of data collection. The postcard will include information about the study, and will describe the various ways to complete the interview. The postcard will also include the website address for the project, and the sample member’s unique username and password for the site. The postcard will be folded and secured with a mailing tab so personal information cannot be viewed until the tab has been broken.

Intensive in-house tracing. The goal of intensive tracing is to obtain a telephone number at which the sample member can be reached so that field interviewing will not be required. Tracing procedures may include (1) Directory Assistance for telephone listings at various addresses, (2) criss-cross directories to identify (and contact) the neighbors of sample members, (3) calling persons with the same unusual surname in small towns or rural areas to see if they are related to or know the sample member, and (4) contacting the current or last known residential sources such as the neighbors, landlords, and current residents of the last known address. Other more intensive tracing activities could include (1) database checks for sample members, parents, and other contact persons, (2) credit database and insurance database searches, (3) drivers’ license searches through the appropriate state departments of motor vehicles, (4) calls to colleges, military establishments, and correctional facilities to follow up on leads generated from other sources, (5) calls to alumni offices and associations, and (6) calls to state trade and professional associations based on information about field of study in school and other leads.

Field tracing and interviewing. One of the challenges presented by the BPS:04/09 full-scale data collection efforts is the need for in-person tracing and interviewing nationwide. We will use a two-tiered tracing strategy for field cases that could not be completed through either self-administered web interview or CATI. Using the best available address for the nonresponding sample members, the cases will be clustered into geographic areas. At that time, field interviewers will be assigned areas with high concentration of sample members (e.g., a major metropolitan area). These field interviewers will be assigned to locate and interview the sample members residing in that cluster. Cases in areas without assigned field interviewers (e.g., cases not clustered with other cases) will be assigned to receive additional intensive tracing. Cases where additional telephone contact information is collected will be returned to data collection by telephone.

2. Web-based Data Collection

The self-administered web-based data collection effort will begin immediately after the initial postcards are distributed to sample members, in late February or early March 2009. This will enable the sample cohort to access the BPS:04/09 website to complete the survey online. The web-based data collection will be enabled throughout data collection for the study, and we estimate that approximately 65 percent of the BPS:04/09 interviews will be completed in this mode. To provide assistance to sample members who choose the self-administered web option, a help desk will be available during the data collection period. Help desk staff will complete the interview by telephone, if requested, when a respondent calls in for assistance.

Halfway through the early response period, we intend to prompt prior round nonrespondents, reminding them of the end date of the early response phase. Prompting has shown mixed results. In the BPS:04/09 field test, it showed no effect, possibly because, first, so many sample members had already completed and, second, there were a limited number of successful prompts (i.e., talked to the sample member, left a message with another person, left a message on an identified answering machine). In contrast, prompting was shown to have raised the nonrespondent participation rate to that of respondents during the BPS:04/06 field test. The major advantage of prompting is that it provides early information that telephone numbers are obsolete, allowing RTI to begin intensive tracing sooner, with little additional cost to the contract since the staff doing the prompting are already working the Help Desk.

3. Telephone Data Collection

All cases not completed by self-administration within 4 weeks of the start of data collection will be eligible for telephone interviewing. The data collection effort will involve locating, contacting, and then interviewing the sample cohort members who have not already responded to the web-based self-administered interview. As noted earlier, data collection for the full-scale study will include both respondents and nonrespondents from the NPSAS:04 and BPS:04/06 studies. Members of the sample who do not respond to the BPS:04/09 self-administered interview or attempts to conduct a telephone interview will receive special attention to ensure that appropriate response rates are achieved.

4. Field Data Collection

The costs of field tracing and interviewing nonresponding sample members are very high, especially relative to self-administered data collection. As a result, only a subset of nonrespondent cases will be sent to RTI field interviewers for in-person contact attempts. Clustering is proposed to reduce field expenses (such as interviewer travel time and other expenses). Cases will be assigned to the field on a flow basis, beginning several weeks after the start of telephone data collection.

5. Refusal Aversion and Conversion

Recognizing and avoiding potential refusals is critical to maximize the response rate. We will emphasize this and other topics related to obtaining cooperation during data collector training. Supervisors will carefully monitor interviewers during the early days of CATI data collection and provide retraining as necessary. In addition, supervisors will review daily interviewer production reports produced by the CATI system to identify and retrain any data collectors with high refusal rates.

Whenever a refusal is encountered, the data collector will enter comments into the CATI record. These comments will include all pertinent data regarding the refusal situation, including any unusual circumstances and any reasons given by the sample member for refusing. Supervisors will review these comments to determine what action should be taken with each refusal. Refusals and partial interviews will require supervisory review and approval prior to being finalized.

There may be instances in which sample members state an unwillingness to complete a telephone interview because of concern for their cell phone usage and charges for the minutes that will accrue. RTI will provide instructions to interviewers on how to respond to sample members who refuse to participate out of concern for their cell charges.

If there is a clear indication that follow-up would be inappropriate (e.g., there are extenuating circumstances, such as illness or the sample member clearly and firmly requested that no further contact be made), the case will be coded as final and will not be recontacted. If the case appears to be a “soft” refusal, follow-up will be assigned to a member of a special refusal conversion team made up of interviewers who have proven to be especially adept at dealing with refusals.

Refusal conversion efforts will be delayed for at least one week in order to give the respondent some time after the initial refusal. Conversion attempts made too soon are often more difficult for the second interviewer. We will not attempt refusal conversion efforts with individuals who become verbally abusive or who threaten to take legal or other action. Refusal conversion efforts will not be conducted to a degree that would constitute harassment. We will respect a sample member’s right to decide not to participate and will not impinge this right by carrying conversion efforts beyond the bounds of propriety.

Incentives to convert refusals, difficult and unable-to-locate respondents. As described in the justification section (section A), we propose to offer incentive payments to all members of the sample population. During the early response phase of data collection, sample members will be paid a $30 incentive for completing the interview. During the production phase of data collection, sample members will be paid a $20 incentive for completing the interview. As discussed further below, this additional production phase incentive is proposed because BPS:04/09 field test results showed that sample members who were paid an incentive during the last interview showed low response rates when no incentive was offered.

In addition, we propose to offer $30 incentive payments to nonresponding members of the sample population. We believe that nonrespondents will be comprised of three primary groups: individuals who have refused to participate in web/CATI interviewing, persons who have proven difficult to interview (i.e., persons who repeatedly break CATI appointments), and sample members who cannot be located or contacted by telephone. Our approach to maximizing the response of these persons, and thereby limiting potential nonresponse bias, involves an incentive payment as a token of appreciation for respondent time and attention.

In the last two weeks of the BPS:04/09 field test, a $40 incentive payment was offered all remaining nonrespondents in the nonresponse phase, replacing the $30 that was offered initially as a nonresponse incentive. The number of completed interviews per day had dwindled to 1 or 2, and the completion rate among those reaching the nonresponse phase was only 43 percent. We expect that the higher incentive amount will be unnecessary for full-scale data collection. Hence, there will not be a $40 incentive in the BPS:04/09 full-scale study. The field test sample was largely comprised of supplemental sample members – that is, those added to the field test sample without having participated in the NPSAS base year interview. Given the slowed progress of data collection at the time, we felt the higher amount of incentive was necessary to encourage response from an already difficult group. The BPS full-scale sample does not contain a similar subset of sample members whose interview behavior is like that of the supplemental sample.

Any respondents who were prior round (BPS:04/06) nonrespondents will be paid an additional incentive amount to compensate for the additional burden of providing background information otherwise collected during the base-year interview. We will offer a $20 differential above the incentive amount for which they would otherwise be eligible. For example, a sample member who was a nonrespondent to BPS:04/06 who completed during the early response phase of BPS:04/09 would be eligible for the $30 early response incentive plus the $20 differential for prior round nonresponse status for a total of $50.

4. Tests of Procedures and Methods

The BPS:04/09 field test forms clearance package submitted in 2007 described four areas of data collection believed to affect overall study participation that would be evaluated during the BPS:04/09 field test data collection. These areas were (1) visibility of mailout materials, (2) notification by cell phones/text messaging, (3) prepaid incentives, and (4) nonresponse conversion incentives. This section revisits those plans and describes the results obtained during the field test.

1. Visibility of Mailout Materials

During the field test, we tested the impact of the visibility of mailout materials on participation rates. Prior to the start of data collection, the field test sample was randomly assigned to two groups: one group received the initial study materials via regular mail in a larger 9 inch by 12 inch envelope, and the other group received the same materials via Priority Mail envelope. The initial mailing contained information about the study, including how to log on to complete the interview.

Results of the mailout experiment were measured in terms participation rates at the end of the early response period for these two groups to determine whether participation was greater for those who received the larger envelopes via regular mail. As shown in exhibit 1, only participation rates among one sample type was significantly higher among the Priority mail packages when compared to rates among regular mail packages. For the sample overall, the difference was not statistically significant.

As described in the field test forms clearance package, the supplemental sample was created when the yield from the base year data collection (NPSAS:04 field test) was inadequate to allow a sufficient test of procedures to be conducted. The majority of the field test sample was the supplemental sample and, therefore, the results of all field test experiments need to be interpreted cautiously. Because they were not part of the base year sample nor were they contacted or interviewed during the base year, the supplemental sample behaved much more like first follow-up respondents than the other large subgroup, those who were both NPSAS:04 and BPS:04/06 respondents. They, therefore, were unlikely to show the same participation that longitudinal sample members have (although not statistically significant, the mailout results for those who participated in both prior interviews are opposite those of the supplemental sample).

Exhibit 1. BPS:04/09 field test early completion rates, by type of mailing and sample type

Because, with the small sample sizes, the results of the mailout experiment were not robust. However, results from the NPSAS:08 field test showed that U.S. Priority Mail envelopes were more effective than regular envelopes in obtaining interviews during the early response period. Hence, we plan to use the U. S. Priority Mail envelopes during the BPS full-scale data collection. The price of the envelopes and mailing is low relative to the cost of locating and attempting to contact sample members with professional staff.

2. Use of Cell Phone Calling and Text Messaging

The use of cell phone calling and text messaging is a relatively new means for contacting sample members. Little research has been conducted on the effects of text messaging on participation rates and it was our intention to use text messages as an additional method for contacting sample members then compare participation rates. However, there was an insufficient number of sample members willing to be contacted via text messaging to allow comparison. Given the interviewers’ observations that, because of minutes charges and battery life, sample members were reluctant to use their cellular phones, we do not feel that text messaging should be pursued for the full-scale data collection.

3. Use of Prepaid Incentives

There has been some evidence which suggests that the use of prepaid incentives increases survey response more than promised incentives alone. The BPS:04/09 field test compared the effect on early interview completion rates of two incentive conditions: (1) a $5 cash prepaid incentive with the promise of a $25 check upon interview completion and (2) a promised incentive of $30 paid in a check.

The results of the field test showed no indication that sample members who received $5 in cash as a prepaid incentive were more likely to complete the BPS:04/09 interview within the first four weeks of data collection, the early response phase. Almost 40 percent of sample members offered a promised incentive completed during the early response phase, while almost 35 percent of those sent $5 with the promise of a $25 incentive completed early. The difference was not statistically significant. Given that there does not appear to be a difference between the two incentive conditions, we plan to offer only promised incentives during full-scale data collection.

When looking at specific sample groups, the results of the incentive experiment did show that 40.3 percent of supplemental sample members offered a promised incentive completed the interview during the early response period compared to 33.9 percent of supplemental sample members offered a prepaid incentive of $5 cash (n=392 per condition; z=1.85; Pr=.06). Among base year/first follow-up respondents, 44.4 percent of those offered a promised incentive completed during the early response period compared to 47.8 percent offered a prepaid cash incentive (n=115 per incentive condition; z=-.53; Pr >.05). Neither comparison was statistically significant.

4. Incentives during Production Interviewing

No incentive was offered or paid during production interviewing. Overall, the response rate observed during the production interviewing phase (14 percent) of the field test was lowest among the phases (early response [51 percent; Z = −7.22; Pr < .05] and nonresponse conversion [34 percent; Z = −3.98; Pr < .05]). Although not showing statistically significant differences (only about 110 sample members responded during the production phase of the BPS:04/09 field test), the response rates shown in table 12 suggest that sample members were not inclined to respond during the production phase of interviewing, which paid no incentive, when they had received an incentive in the prior interview, BPS:04/06. The exception was the group who received no production incentive during the BPS:04/06 field test suggesting that those paid for the prior interview expect an incentive again; those not paid do not have such an expectation. Therefore, because all production phase respondents during the BPS:04/06 full-scale were incentivized, we propose to incentivize them again for the BPS:04/09 data collection.

Table 12. BPS:04/09 field test response rates during production phase of interviewing, by BPS:04/06 incentive amount paid

BPS:04/06 incentive paid	Observed response rate in BPS:04/09 FT production phase (percent)
Early response $30	9.0
Production $0	14.5
Production $20	11.0
Nonresponse conversion $30	7.4

5. Reviewing Statisticians and Individuals Responsible for Designing and Conducting the Study

Names of individuals consulted on statistical aspects of study design along with their affiliation and telephone numbers are provided below.

Name Affiliation Telephone

Dr. Sara Wheeless RTI 919/541-5891

Dr. Jennifer Wine RTI 919/541-6870

Ms. Melissa Cominole RTI 919/990-8456

Dr. Karol Krotki RTI 202/728-2485

Mr. Peter Siegel RTI 919/541-6348

Dr. Lutz Berkner MPR 510/849-4942

In addition to these statisticians and survey design experts, the following statisticians at NCES have also reviewed and approved the statistical aspects of the study: Dr. Tom Weko, Dr. James Griffith, Dr. Tracy Hunt-White, Ms. Linda Zimbler, and Mr. Ted Socha.

6. Other Contractors’ Staff Responsible for Conducting the Study

The study is being conducted by the Postsecondary Longitudinal and Sample Survey Studies unit of NCES, U.S. Department of Education. NCES’s prime contractor is RTI. RTI is being assisted through subcontracted activities by MPR Associates. Principal professional staff of the contractors not listed above, who are assigned to the study, are provided below:

Name Affiliation Telephone

Ms. Donna Anderson RTI 919/990-8399

Mr. John Doherty RTI 919/541-7120

Ms. Vicky Dingler MPR 510/849-4942

Ms. Kristin Dudley RTI 919/541-6855

Mr. Jeff Franklin RTI 919/485-2614

Mr. Joe Simpson RTI 919/541-5941

1An indicator of Title IV eligibility has been added to the analysis files from earlier NPSAS studies to facilitate comparable analyses.

2 Cominole, M., Siegel, P., Dudley, K., Roe, D., and Gilligan, T. 2004 National Postsecondary Student Aid

Study (NPSAS:04) Full Scale Methodology Report (NCES 2006–180). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

3 A “potential FTB” is one who is expected to have been a first-time beginning student during the “NPSAS year” (July 1, 2003–June 30, 2004) but who was not confirmed as such during the student interview. Students were identified as potential FTBs by their sample institution. Other data sources (CPS, CADE) also provide an indication of FTB status for the time period of interest.

4 The results from the advance tracing for BPS:04/06 were used to determine whether a student had been located. The National Change of Address file (NCOA) was used to obtain updated addresses for the student, then Telematch was used to obtain an updated telephone number. The student was classified as “located” if Telematch either returned a new telephone number or confirmed the current telephone number.

BPS:04/09 Supporting Statement Request for OMB Review (SF83i) 15