Att_Perkins 5 Supporting Statement Part B

Part B: Collection of Information Employing Statistical Methods

This submission requests clearance for the data collection instruments developed as part of the Statutorily-mandated independent evaluation and assessment of career and technical education programs under Perkins IV, referred to as the National Assessment for Career and Technical Education (NACTE). This study will collect survey information and fiscal allocation data from state directors of all eligible agencies and a sample of administrators of eligible recipients. This section describes the design for selecting a sample of local program administrators to participate in the evaluation.

B.1. Potential Respondent Universe and Sample Selection Method

Survey and Fiscal Allocation Data: State Directors of CTE

Since we plan to collect survey information and fiscal allocation data from all state directors of CTE within eligible agencies, there is no sampling plan to discuss with regard to this data collection activity.

Survey Data: Eligible Recipients (Secondary and Postsecondary Local Program Directors)

The sampling frame for the NACTE will be constructed from the Common Core Data (CCD) file and the Integrated Postsecondary Education Data System (IPEDS). The CCD will cover LEAs. The target population for the LEA survey will consist of those agencies that offer 12^th grade instruction and are located within the 50 states, the District of Columbia, Puerto Rico, or the Virgin Islands. Bureau of Indian Affairs (BIA) LEAs also are included. Table 5 shows the population and sample sizes for the LEAs. The sample has been allocated proportionally to each of the sampling strata that are formed from the cross-classification of poverty level and LEA size and rounded up to the next integer.

Table 5 Population and Sample Counts for LEAs, Excluding Area CTE Centers, by Strata
	Small LEA		Medium LEA		Large LEA		Unknown		Total
Poverty Level	Popu-lation	Sample	Popu-lation	Sample	Popu-lation	Sample	Popu-lation	Sample	Popu-lation	Sample
Low Poverty	546	87	1,588	253	1,057	168	0	0	3,191	508
Medium Poverty	1,308	208	2,587	413	1,155	184	0	0	5,050	805
High Poverty	1,200	191	1,600	255	986	157	0	0	3,786	603
Unknown	207	33	89	14	22	4	175	28	493	79
Total	3,261	519	5,864	935	3,220	513	175	28	12,520	1,995
Source: Common Core of Data, 2006-07. Notes: LEA with 499 students or less were classified as small LEA; those with 500 to 2,999 students were classified as medium LEAs; and those with 3,000 or more students were classified as large. Low-poverty LEAs are those with less than 25 percent of students; medium-poverty LEAs those with between 25 and 50 percent of students; and high-poverty LEAs those with more than 50 percent of students receiving free or reduced price lunches.

Among LEA, there is a subset of institutions, termed area CTE centers, which provide specialized CTE services on a part-time basis to students who receive most or all of their academic instruction at their home high school. These area CTE schools typically serve multiple high schools, and most often are administered as a separate school within a given LEA. There are, however, a handful of these centers that function as an independent LEA. According to the most current data that exist, there were a total of 1,191 area career and technical education schools in the United States in 2002, and of these, 48 operated as independent LEA.¹ To gather data on these specialized LEAS, the NACTE study team will sample with certainty the universe of local program directors in these LEAs.

Accordingly, the total secondary program director survey will be based on 2,043 LEAs (1,995 LEAs and 48 area CTE centers).

The IPEDS will cover IHEs. The target population for the survey will consist of public less-than- two-year postsecondary institutions, public two-year colleges, and tribally controlled colleges. Public four-year colleges, area or regional schools funded with postsecondary resources, adult schools, private non-profit colleges, and other not-identified institutions are excluded.² Table 6 shows the population and sample sizes for the included IHEs. The sample has been allocated proportionally to each of the sampling strata that are formed from the cross-classification of financial need level and IHE type and rounded up to the next integer, except for Tribal 2-year and 4-year colleges that fall in the low financial need level category which will be sampled with certainty.

Table 6 Population and Sample Counts for IHEs, by Strata
	Public less than 2-year institutions		Public 2-year colleges		Tribal 2-year and 4-year colleges		Total
Level of Financial Need	Population	Sample	Population	Sample	Population	Sample	Population	Sample
Low	28	20	189	134	3	2	220	156
Medium	133	94	872	617	12	8	1,017	719
High	61	43	62	44	17	12	140	99
Unknown	43	30	3	2	0	0	46	32
Total	265	187	1,126	797	32	22	1,423	1,006
Source: Integrated Postsecondary Education Data System (IPEDS): 2007-08; 2006-07.
Notes: Level of financial need represents the percentage of first-time, full-time students receiving federal grant aid within an IHE compared to the total population of first-time, full-time students within the IHE. Low financial need IHEs are those with less than 25 percent of students; medium financial need IHEs are those with between 25 and 75 percent of students; and high financial need IHEs are those with more than 75 percent of students receiving federal grant aid.

B.2. Procedures for the Collection of Information

Statistical Methodology for Stratification and Sample Selection

Each year, the federal government provides a state allocation of Perkins IV Title I funds to each state, using a formula contained in the statute.³ States must allocate at least 85 percent of these resources to LEAs and IHEs using a formula that takes into account characteristics of provider populations. At the LEA level, 30 percent of federal Title I funds are allocated based on the number of youth ages 5 through 17 residing in an LEAs boundaries and 70 percent based on the number of youth ages 5 through 17 who are from families below the poverty line, compared to the total number of youth within each category who reside within LEAs statewide.

Generally, postsecondary Title I funds are allocated based on the number of individuals attending an eligible institution who are Pell Grant recipients (or recipients of assistance from the Bureau of Indian Affairs) relative to the total number of such individuals who are enrolled within eligible institutions throughout the state.

To encourage the most efficient use of funding, Perkins IV requires that LEAs and eligible institutions achieve a threshold level of funding to qualify for a grant ($15,000 for LEAs and $50,000 for eligible institutions, respectively). Recognizing that smaller LEAs and eligible institutions might have difficulty achieving these funding thresholds, Congress provided for flexibility in the legislation. LEAs and eligible institutions failing to achieve the minimum funding level may pool their eligibilities by forming a consortium or states may waive the minimum funding threshold for LEAs or eligible institutions that can demonstrate they meet certain conditions.

A stratified simple random probability sample will be selected to be representative of all LEAs and eligible institutions, excluding area CTE centers.⁴ Researchers will stratify the sample at both the secondary and postsecondary levels using the same criteria used to allocate Perkins IV funding by eligible agencies. Specifically, at the secondary level, the LEA population will be stratified based on agency K–12 membership and the percent of K–12 membership who are free lunch eligible or reduced lunch eligible students.⁵ To ensure that the sample distribution looks like the population distribution, the sample was allocated proportionally to the sample strata where the proportion was based on the population proportion. Researchers will sample with certainty the 48 area CTE centers that operate as standalone facilities.

Although all LEAs enrolling students may qualify for a Perkins IV allocation, resources are typically distributed to those offering secondary level program services (i.e., grades 9–12). For this reason, the population of LEAs included in this study is limited to those offering 12^th grade instruction.⁶ Instructional services within LEAs may be offered in comprehensive high schools, in CTE high schools or area regional education centers, in LEAs operating as a stand-alone regional education service agency or within a state, federal or other service agency.⁷

The postsecondary population will be stratified by institutional type and the percentage of federal grant aid recipients within each eligible institution.⁸

Estimation Procedure

To account for the complex survey design and differential weighting, point estimates and standard errors will be calculated using software that will support descriptive statistics of categorical and continuous variables (i.e., the primary estimates that will be calculated from the survey data).

Degree of Accuracy Needed for the Purpose Described in the Justification

Secondary Agencies (LEAs, excluding area CTE centers) - Nonresponse. Assuming that the population proportion under the null hypothesis is 0.5000, a required sample size of 1,136 LEAs from a population of 12,520 LEAs achieves 81% power to detect a difference of 0.04 using a two-sided binomial test. The target significance level is 0.0500, and the actual significance level achieved by this test is 0.0499.⁹ To account for anticipated non-response, the required sample size was increased. Based on the expected 85 percent response rate for LEAs, the non-response adjusted sample size was increased to 1,337 LEAs. Also, approximately two-thirds of LEAs were actually awarded a grant. To account for anticipated ineligible LEAs, the non-response adjusted sample size was increased. Based on the expected 67 percent eligibility rate for LEAs, the final sample size was increased to 1,995 LEAs.

Eligible area CTE centers - Eligibility. Recall that a total of 48 centers were identified as standalone facilities. Sampling with certainty on these institutions provides a final sample size of 48.

Eligible Institutions (postsecondary) - Nonresponse. Assuming that the population proportion under the null hypothesis is 0.5000, a required sample size of 684 IHEs from a population of 1,423 IHEs achieves 81% power to detect a difference of 0.04 using a two-sided binomial test. The target significance level is 0.0500, and the actual significance level achieved by this test is 0.0438.¹⁰ To account for anticipated non-response, the required sample size was increased. Based on the expected 85 percent response rate for IHEs, the non-response adjusted sample size was increased to 805 IHEs. Also, approximately four-fifths of IHEs were actually awarded a grant. To account for anticipated ineligible IHEs, the non-response adjusted sample size was increased. Based on the expected 80 percent eligibility rate for IHEs, the final sample size was increased to 1,006 IHEs.

While all LEAs and eligible institutions in the United States may qualify for Perkins funding, no data currently exist on the actual number that receive CTE grants. Although states are required to report annually on their allocation of funding among educational sectors and administrative programs (e.g., Title I, Title II), states are not required to document to the U.S. Department of Education the number of local grants actually made. The most current statistics on LEA and eligible institutions participation in Perkins comes from the 2004 NAVE study, which suggests that roughly two-thirds of eligible LEAs and four-fifths of eligible institutions were awarded Perkins funds in the 2000–01 program year.¹¹ These data, however, are based on information collected for 39 states and so fail to accurately capture the true participation rates among these providers.

In an effort to identify LEAs and eligible institutions receiving grants in the 2008–09, the NACTE research team consulted staff at the NASDCTEc, the professional organization for state directors, to determine whether the association had information on Perkins IV grants made by eligible agencies. Unfortunately, state data compiled by the organization was incomplete: the organization had records for less than half of all states, with many lacking NCES District IDs and/or IEDS IDs that would permit researchers to access demographic information contained with the CCD and IPEDS.

To economize resources and to minimize data burden for LEAs and eligible institutions that did not receive a grant, researchers will consult the data supplied by the NASDCTEc to determine eligibility without having to contact the secondary agencies.

Fiscal Allocation Data

State directors of CTE will be asked to provide an electronic file detailing their Perkins IV allocations for the 2006-07 and 2009-10 program years. This request, which will be included in the survey notification mailing, will include a letter describing the purpose of the data collection, a template indicating the type of data to be collected and the procedures for reporting it, and a contact number for individuals who have questions about the request.

B.3 Methods for Maximizing Response Rates

Because respondent contacting is the first stage of the study, obtaining the cooperation of as many LEAs and eligible institutions as possible is critical. The goal is to achieve a survey participation rate of at least 85 percent for the three populations being surveyed (i.e., state directors, administrators of LEAs, and directors of eligible institutions) and to complete fiscal allocation data from the entire population of states. Many respondents are familiar with the nature of CTE funding operating through Perkins, and recognize the study’s importance to CTE’s future.

Potential respondents will be sent three mailings that will include a cover letter, credentials for accessing the web-survey (an ID and password), and a study brochure describing the survey and fiscal data collection, including a study summary, outline of the data collection procedures, project schedule, and details regarding the protection of respondent privacy and study confidentiality procedures. The lead letters to respondents will be designed to raise awareness and promote respondent motivation to participate in the survey. Reminder/thank you postcards will be sent to respondents 7–10 days after the initial two mailings. A third and final mailing will also include a hardcopy questionnaire and a return stamped envelope.

Within one week of the second mailing, the research team will begin a series of follow-up calls. At a minimum, reminder calls will be placed at approximately 4, 8, and 12 days after the mailing, to encourage non-responders to participate. For those respondents who believe that the study is overly burdensome, the first contact will provide us with an opportunity to have a senior project staff member address their concerns. Contacting respondents also provides us with an invaluable opportunity to establish initial rapport.

After allowing adequate time for materials to reach the respondents, we will start making follow-up calls to them to secure participation. A core group of staff in RTI’s Raleigh Call Center (RCC) will be identified to carry out these contacts. Each staff member will be assigned a set of respondent directors and/or institutions, which will remain assigned to that person throughout the process. This will allow a specific individual to establish a rapport with the respondent and provide the institution with a reliable point of contact. Staff members will be thoroughly trained to understand basic CTE concepts and to appreciate the complex (and at times ambiguous) CTE education environment, which helps them establish credibility with the institution.

NonResponse Issues

Recognizing and avoiding refusals is critical to maximizing the survey response rate. We will emphasize this and other topics related to obtaining cooperation during training of our Institutional Contacting (IC) staff. Supervisors will monitor ICs intensely during the early days of data collection and provide retraining as necessary. In addition, the supervisors will review daily production to identify and retrain any ICs who are proving to be less effective than their colleagues at contacting institutions and obtaining participation.

Whenever a refusal is encountered, the IC will enter comments into the web-based Institutional Contacting System (ICS) which is used to track progress of all respondents. These comments will include all pertinent data regarding the refusal situation, including any unusual circumstances and any reasons given by the sample member for refusing. Supervisors will review these comments to determine what action should be taken with each refusal. No refusal or partial response will be coded as final without supervisory review and approval. In completing the review, the supervisor will consider all available information about the case and will initiate appropriate action.

If there is a clear indication that follow-up would be inappropriate, the case will be coded as final and will not be recontacted. If the case appears to be a “soft” refusal, follow-up will be assigned to project staff. Refusal conversion efforts will be delayed for at least one week in order to give the respondent some time after the initial refusal. Conversion attempts made too soon are often more difficult. We will not attempt refusal conversion efforts with individuals who become verbally abusive or who threaten to take legal or other action. Refusal conversion efforts will not be conducted to a degree that would constitute harassment. We will respect a sample member’s right to decide not to participate and will not impinge this right by carrying conversion efforts beyond the bounds of propriety.

B.4 Test of Procedures

We will use a combination of survey instrumentation and a fiscal allocation data collection form to collect information from eligible agency and LEA and IHE staff.

Survey Instruments

To finalize survey instruments, pilot tests of the four survey instruments were conducted with individuals selected from states that would provide some diversity based on geography, population size, and funding levels.¹² Individuals also were selected based on their experience and knowledge of the field. To provide internal consistency, a state secondary director and local program provider (Oregon) and state postsecondary director and local program provider (Minnesota) were selected to assess whether cross sector conditions within a given state might affect survey interpretation.

State Director Survey

• 2 former secondary state directors (Oregon and Texas)

• 2 former postsecondary state directors (Minnesota and Oklahoma)

Local Program Director Survey

• 2 acting secondary program directors (Oregon and Wyoming)

• 2 acting postsecondary program directors (Minnesota and New Mexico)

Although each pair of reviewers was responsible for pre-testing a unique survey, some questions are repeated across surveys, enabling researchers to obtain multiple responses from a variety of perspectives.

To guide their review, pre-test volunteers were provided with the following set of questions to which they were asked to provide specific feedback:

1. How long did it take you to complete the survey? How long would you estimate it will take, in general, for your colleagues to complete the survey? (½ hour?, 1 hour?, more?)

2. Did you identify any specific areas that, if changed, might conserve time in completing the survey?

3. In general, is there overall clarity of the survey’s purpose? Is there coherence to the order of questions?

4. Are there any questions where the directions to respond were unclear?

5. Are there any questions where the phrasing or purpose of the question was unclear?

6. Were you able to complete the survey yourself? Or, did you/would you need to access information from another staff member? If so, on which question(s)?

7. Do the questions appear relevant to the implementation of Perkins IV as you understand the Act’s requirements? If not, which question(s) would you change and how? Are there any questions you would delete? Why?

8. Do you have any additional feedback that would improve the quality and flow of the questions?

Information provided by pilot test members was used to make final revisions to the survey instruments. Key changes included (1) incorporating web links into the survey to permit respondents to reference legislative language, (2) refining wording to clarify question meaning, and (3) simplifying reporting requirements for questions requiring information on the proportion of students participating in CTE programs of study. Respondents also suggested that alerting respondents to the types of data that they might be expected to use to complete the survey would save a great deal of time. Accordingly, NACTE researchers will include in the initial contact letter sent to each respondent a list of the type of data that will be needed to complete the survey, along with a recommendation that directors assemble this information prior to logging onto the survey website.

Former state directors reported that the secondary and postsecondary surveys took between 45 to 90 minutes to complete. Local secondary program directors reported that the survey took between 25 minutes to 60 minutes to complete, and postsecondary directors, between 30 minutes and 60 minutes. It was suggested that surveys could require additional time if respondents in states with written state policies needed to attach multiple examples of state policies or website addresses. However, it was felt that, if provided information beforehand, state and local program directors would have time to prepare for the survey, shortening response times.

Fiscal Data Collection Tool

To collect state fiscal allocation data, the study team will supply state CTE directors with a set of instructions detailing the types of data that are to be collected and a requested file format for the submission of electronic data. Researchers will request that the former state directors conducting the survey review (see above) provide feedback on the instructions contained in the fiscal data collection tool. The research team will also share a copy of the instrument with the Program Committee of the National Association for Career and Technical Education Information (NACTEI), which is a national professional organization of CTE data analysts and field representatives dedicated to the development and improvement of CTE information on finance and accountability systems.

B.5 Individuals Consulted on Statistical Aspects of the Design

Information for this study will be will be collected by RTI on behalf of the U.S. Department of Education. Analysis of data will be conducted by MPR and AED. Input on the statistical aspects of the design was obtained from the following individuals:

Department of Education

Michael Fong, Policy and Program Studies Service, COR, NACTE, 202-401-7462

Jay Noell, Policy and Program Studies Service, Director, NACTE, 202-401-1026

National Evaluation Contractors

MPR Associates Inc. Steve Klein, Director, 503-963-3757

Bob Fitzgerald, Senior Research Associate, 510-849-4942

Karen Levesque, Director, 510-849-4942

AED Ivan Charner, Vice President and Director, 202-884-8173

Robin White, Senior Program and Policy Director, 202-884-8254

RTI James Isaac, Survey Director, 919-541-6342

Darryl Creel, Research Statistician, 301-770-8229

1 Data on the total number of area CTE centers was obtained from Table 2.5 contained in the NCES publication Career and Technical Education in the United States: 1990 to 2005 (NCES 2008-035). To identify standalone facilities, researchers compared the names of identified centers within states to the list of centers contained in the Common Core of Data, 2001-02. The 48 area CTE centers identified for this study were listed as independent LEA with their own NCES ID number. Remaining area CTE centers were listed as a school within an LEA. These centers were excluded from the study since information on their operations would be obtained from the program director of the LEA in which they were housed, assuming that the LEA were randomly selected for study participation.

2 While Perkins IV defines eligible institutions to include public or nonprofit institutions of higher education (as defined in Title IV of the Higher Education Act of 1965); LEA and area and technical education schools providing education at the postsecondary level; and BIA-funded colleges or tribally controlled colleges, findings from the 2004 National Assessment of Vocational Education indicate that a disproportionate share of federal postsecondary resources flow to public less-than-two-year postsecondary institutions and 2-year colleges (20.2 percent and 67.6 percent of all postsecondary Perkins funds, respectively). Public 4-year colleges and universities accounted for just 4.3 percent of postsecondary Perkins funds distributed in the 2000-01 program year. Although the 2006 Perkins Act expands federal resource eligibility to include baccalaureate granting institutions, OVAE staff report, based on information collected during state monitoring visits, that the number of 4-year colleges and universities participating in the Act has remained relatively constant over time. Given the low likelihood that a random sample of public 4-year or private non-profit colleges and universities would include Perkins grantees, researchers opted to exclude this group from the study. Researchers also excluded area or regional technical schools funded with postsecondary resources due to difficulties in identifying agencies receiving postsecondary fund.

3 The federal Perkins IV allocation also includes Title II resources, which are allocated by states to LEAs and IHEs using either a competitive process or through a formula developed by the state. These funds constitute less than 10 percent of the total state Perkins IV allocation. Since state allocation criteria used to distribute Title II resources are not available and may differ substantially across states, the sample selection methodology is based on Title I funds distribution across states. States electing not to merge their Title II funds with their Title I funding will be requested to provide separate information on their use of these funds as part of the survey.

4 We considered using probability proportional to size sampling with the measure of size being the number of students at the school, but we were unsure how correlated these two variables would be. There is no information on the frame for the number of students in CTE. This is one of the questions in the survey.

5 High school students qualifying for free or reduced school lunches are likely undercounted since older students are less likely to take advantage of this program. As such, LEAs comprised of high schools alone may qualify for less Perkins IV funding then they might otherwise be eligible. Since this is a systematic bias (i.e., intrastate Perkins IV funds distribution also is affected by this condition), there should be no effect on the sampling frame.

6 Although the law permits funds to be spent on CTE services in seventh or eighth grade, less than 1 percent of Perkins grants were allocated at the middle school level in 2000-01, and, as such, are not included in this analysis.

7 Data on the number of area CTE schools located within states is available for 2002 through the CCD. Due to data reporting limitations, information contained within the database do not accurately capture the total number of schools operating nationwide, much less the number within LEAs. Statistics published in the NCES report, Career and Technical Education in the United States: 1990-2005, which use CCD data supplemented with a review by state directors of CTE, suggest that there were roughly 1,200 area career technical centers operating in the 2002. When sampling, researchers will seek to ensure that a representative group of LEAs operating these specialized instructional facilities are included.

8 IPEDS does not distinguish Pell grant recipients from other recipients of financial aid and is limited to first-time full-time freshmen. In practice, IHEs report the number of their Pell grant and Bureau of Indian Affairs grant recipients to the state, which distributes resources based on submitted data. These data are not submitted by states to the federal government, and no record of these data are available. Use of IPEDS provides the best standardized, national data on the percentage of students within IHES that are eligible for student aid.

9 Hintze, Jerry (2008). PASS 2008. NCSS, LLC. Kaysville, UT. www.ncss.com

10 Hintze, PASS 2008.

11 Eligible secondary recipients were defined as LEAs and area vocational schools that offered at least 10^th, 11^th, or 12^th grade education, either as independent LEAs or as part of a consortium, based on the CCD. The estimate for postsecondary institutions was based on IPEDS and excludes 4-year colleges and universities and private non-profit institutions. Therefore, this estimate likely overstates the actual number of postsecondary grantees.

12 Since the pilot study involved less than 9 individuals in total (i.e., 4 reviewers for the state director and 4 reviewers for the local survey), OMB approval was not requested prior to survey administration.

MPR Associates Inc. 29