OMBSectionB_11-29

Contract No.: ED-04-CO-0112

MPR Reference No.: 6208-510

Request for OMB Clearance of Data Collection for the Child Care Access Means Parents in School (CCAMPIS) Program

Revision of Currently Approved Collection (OMB #1875-0242)

Section B

November 29, 2007

Submitted to: U.S. Department of Education Policy and Program Studies Service 400 Maryland Ave. SW, Room 6W226 Washington, DC 20202 Project Officer: Patricia Butler, Ph.D.

Submitted by: Mathematica Policy Research, Inc. 600 Maryland Ave. S.W., Suite 550 Washington, DC 20024-2512 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 Project Director: Wendy Mansfield, Ph.D. Survey Director Kirsten Barrett, Ph.D.

CONTENTS

Section Page

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS 20

1. Respondent Universe 20

2. Procedures for Sampling Methods and Analysis 23

3. Methods To Maximize Response Rates 31

4. Tests of Procedures or Methods 32

5. Individuals Consulted on and Responsible for Statistical Design 33

APPENDIX A: detailed list of changes to the phase I survey a-1

appendix B: CHILD CARE SURVEY OF POSTSECONDARY INSTITUTIONS B-1

APPENDIX C: LEGISLATION SUPPORTING CCAMPIS STUDY C-1

APPENDIX D: SUMMARY TABLE OF SURVEY ITEMS AND QUESTION–BY-QUESTION JUSTIFICATIONS D-1

APPENDIX E: LETTERS TO RESPONDENTS E-1

APPENDIX F: CONFIDENTIALITY PLEDGE F-1

APPENDIX G: Phase I mEMO G-1

APPENDIX H: REFERENCES h-1

TABLES

Table Page

B-1 NUMBER OF CCAMPIS INSTITUTIONS BY YEAR 20

B-2 NUMBER OF CCAMPIS GRANTEE INSTITUTIONS BY STATE OR Territory, FISCAL YEARS 2001 AND 2002 22

B-3 SAMPLE SIZES AND MDD BETWEEN CCAMPIS GRANTEE AND SIMILAR NONGRANTEE INSTITUTIONS 29

B-4 POPULATION PROPORTION IN CCAMPIS GROUP, MDD, AND POPULATION PROPORTION IN THE COMPARISON GROUP 30

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

As background, this section describes the construction of the sampling frame and the sample selection procedures (which include matching CCAMPIS and non-CCAMPIS institutions) that were carried out before implementing the Phase I data collection. We also discuss the expected precision of the estimates. Data from Phase I (for items that were retained) and Phase II will be combined for analysis.

1. Respondent Universe

CCAMPIS institutions are defined as Title IV postsecondary institutions that received CCAMPIS grants during the four cohorts of the program that were funded in fiscal year (FY) 1999, FY2001, FY2002, and FY2005. To be eligible for CCAMPIS grants, postsecondary institutions must have received at least $350,000 in Pell Grant funds in the previous fiscal year. A total of 576 CCAMPIS grants have been awarded since 1999. Table B-1 provides the number of CCAMPIS institutions by year.

TABLE B-1
NUMBER OF CCAMPIS INSTITUTIONS BY YEAR

Yeara	Count
1999	85
2001	222
2002	122
2005	147

Source: Lists of CCAMPIS grantees provided by U.S. Department of Education.

^aThere was no competition in 2000, 2003, and 2004.

The population of interest for the study will include two groups: (1) postsecondary institutions that received CCAMPIS grants in 2001 and 2002 and (2) CCAMPIS-eligible postsecondary institutions that did not receive such grants.

CCAMPIS Institutions. The cohorts of postsecondary institutions awarded CCAMPIS grants during FY2001 and FY2002 were identified as the population of interest for the following reasons:

• The 2001 and 2002 grantees will have received up to four years of grant funding for their child care services and therefore will have had an opportunity to implement and refine their services and form perceptions of service effectiveness in promoting students’ persistence and degree completion.

• When the survey is fielded at the beginning of the 2007–2008 school year, the 2005 grantees will have implemented only two years of CCAMPIS grant funding; the services implemented by these grantees may not reflect the full capacity of four years of CCAMPIS grants; and staff will not have had an opportunity to observe potential effects of the services.

• The CCAMPIS program for the 1999 grantees differed from the CCAMPIS program in later rounds: the CCAMPIS program was not housed in the service area that had responsibility for administering the Federal TRIO Programs, the 1999 grantees were less likely to have had a child care program located on campus, and the amount of grant funds awarded to institutions was much smaller. For these reasons, the changes that the 1999 grantees were able to implement may not be typical of the services offered by later grantees.

For this study, the sample includes all 352 institutions in the sampling frame of CCAMPIS institutions for 2001 and 2002.¹

Restricting the population to CCAMPIS grantees in FY2001 and FY2002 means that the study results will pertain to only the 352 institutions in those cohorts (see Table B-2 for the distribution of the CCAMPIS population by state). For this reason, the results will be generalizable only to CCAMPIS grantee institutions that have had a grant for four years.

TABLE B-2

NUMBER OF CCAMPIS GRANTEE INSTITUTIONS BY STATE OR Territory,
FISCAL YEARS 2001 AND 2002

State	Count
Alabama	7
Arizona	7
Arkansas	2
California	49
Colorado	8
District of Columbia	1
Florida	13
Georgia	10
Idaho	3
Illinois	24
Indiana	6
Iowa	7
Kansas	3
Kentucky	7
Louisiana	6
Maine	4
Maryland	4
Massachusetts	3
Michigan	10
Minnesota	4
Mississippi	3
Missouri	7
Montana	4
Nebraska	4
Nevada	1
New Jersey	5
New Mexico	2
New York	17
North Carolina	11
Ohio	12
Oklahoma	4
Oregon	4
Palau	1
Pennsylvania	24
Puerto Rico	4
South Carolina	5
South Dakota	2
Tennessee	5
Texas	23
Utah	5
Virginia	5
Washington	14
West Virginia	4
Wisconsin	6
Wyoming	2
Total	352

Non-CCAMPIS Institutions. As noted, the population of non-CCAMPIS institutions is defined as CCAMPIS-eligible Title IV institutions that have never received CCAMPIS funding. The data source for constructing the study population is the IPEDS database. Information on the amount of an institution’s Pell Grant funds awarded to students for the preceding fiscal years (i.e., FY2000 and FY2001) is available from the IPEDS Finance Data component.

2. Procedures for Sampling Methods and Analysis

The study sample includes the universe of all postsecondary institutions that received CCAMPIS grants in FY2001 and FY2002. In addition, we selected a sample of eligible non-CCAMPIS institutions for use as comparison institutions. Unlike the case of a regular sample survey study in which the sample provides the basis for generalizing about a larger group/population, the present study focused on the selection of non-CCAMPIS institutions in order to generate a set of comparison institutions that “match” the 352 CCAMPIS institutions described earlier. The matched comparison group of institutions ensures that analysis or comparison of CCAMPIS and non-CCAMPIS groups is not subject to selection bias² (or at least minimizes any such bias).

Using Propensity Score Models to Identify the Comparison Group. From the population of eligible non-CCAMPIS institutions, we used the Propensity Score Matching (PSM) method to select a sample of institutions that are comparable to or “match” the 352 CCAMPIS institutions. The PSM method estimated propensity scores based on several observed characteristics on which the two groups (CCAMPIS and eligible non-CCAMPIS institutions) were matched later. We estimated propensity score models using the logistic regression method, whereby the binary variable that indicates status as a CCAMPIS or eligible non-CCAMPIS comparison group member was regressed on a set of predictors. For the PSM predictors, it was important to include institutional, student, and community characteristics as well as state child care policies. Differences in these characteristics may affect the supply of child care services in the community and state and the demand for these services at postsecondary institutions. The IPEDS database provided institution- and student-level characteristics; 2006 demographic data aggregated across block groups was the key community-level matching variables;³ and state child care policy data were gathered from several other sources. These other sources include Child Care Bureau (U.S. Department of Health and Human Services) statistics available on the Web as well as state information compiled by Schulman and Blank (2005) and by the National Association of Child Care Resource and Referral Agencies. The following are examples of each of the four types of characteristics considered as matching variables:

1. Institutional characteristics. Type (two- or four-year), control (public or private), and size of institution; whether the institution offers on-campus child care; and financial data, such as educational and general expenditures.

2. Student characteristics. Number of part- and full-time students; number of Pell Grant recipients (and their dependent status); and whether the campus is residential or commuter, based on the number of students living on campus.

3. Community characteristics aggregated across block groups. Percentage of population by age and race and percentage of households by income group..

4. Child care policy (at state level). Indicators of the availability of state child care assistance for low-income families (for example, percentage of eligible children receiving child care subsidies, percentage of subsidized children served by child care centers, and whether the state has a waiting list for child care assistance); whether child care subsidy eligibility covers education activities and under what circumstances (whether parents must also be working, hours of work required per week, and maximum number of years of education or highest level of degree allowed); and indicators of the cost of child care in the state (including average annual fees paid for full-time care for infants, preschool children, and school-age children and copayments for families receiving child care assistance). We also will consider the geographic location of the matched institutions (for example, states or regions). For this purpose, we may carry out matching within a region or include “region” as a predictor in the PSM model.

It is essential to note that we first performed exploratory data analyses on the above variables to determine whether they were predictors of receipt of a CCAMPIS grant in the PSM model.

The goal of PSM was to identify one non-CCAMPIS institution similar to each CCAMPIS institution so that 352 similar non-CCAMPIS institutions would be available for comparison. In deciding which of the alternative methods of PSM should be used, we determined the extent of overlap between the estimated propensity scores for the CCAMPIS institutions and those for the eligible comparison institutions. Matching was based on the caliper technique,⁴ where the caliper was set to be a small number (Rosenbaum and Rubin 1985). For each grantee institution, this method selected all potential comparison institutions whose propensity scores fell within a specified range, or “caliper,” of the grantee’s propensity score. In this study a match for each grantee institution was determined as the closest non-grantee institution; that is, the non-grantee institution with the smallest absolute difference of propensity scores. When the match could not be found within the specified caliper, a larger caliper was used.

To assess the covariate balance between the CCAMPIS and non-CCAMPIS institutions before and after matching, we computed descriptive statistics (using means or proportions⁵) separately for each covariate for both groups. MPR then performed statistical tests that assessed whether the two groups were different or similar in terms of the distribution of the covariates.

Subsample for Phase II Data Collection. The full sample was stratified by control of the institution (public, private nonprofit, private for-profit) and level of the institution (four or more years, at least two but less than four years), so that there were six sampling strata. About 10 percent of the matched CCAMPIS grantee and non-grantee institutions were selected randomly within each stratum for the Phase I sample (38 pairs or 76 institutions). The remaining 90 percent of the sample will be included in Phase II (314 pairs or 628 institutions).

Statistical Power and Expected Precision of the Full (Phase I and Phase II) Sample. Responses from the Phase I and Phase II samples will be combined. The degree of accuracy of estimates for the full sample is illustrated through a statistical power analysis under the assumption that the respondents are a random sample of the population. We performed a prospective power analysis based on a fixed sample size, confidence level, and power of the test in order to determine the level of precision of the resulting estimates and the magnitude of the CCAMPIS effect that is detectable.

We used the following assumptions in the power analysis:

• The study is designed to detect effects with a confidence level of 90 percent (corresponding to type-I error percent) and power of 80 percent.

• To maintain a reasonable level of precision for statistical analyses, the sample design includes 352 institutions that received CCAMPIS grants in 2001 (228 institutions) or 2002 (124 institutions). We selected 352 comparison institutions that match the CCAMPIS institutions (one-to-one matches), since comparison with a balanced sample size has more power than one with an imbalanced sample size, and since identifying at least one comparable institution per CCAMPIS institution will allow us to detect a minimum difference of at least 10 percentage points between CCAMPIS and non-CCAMPIS institutions (assuming 80-percent power and a 90-percent confidence level), on percentages calculated from survey responses.

• Nonresponse will exist; an estimated 85 percent of the institutions will respond to the survey.

• Nonrespondents may have different characteristics than respondents who complete the surveys. Therefore, analyzing the data based only on completed cases may introduce bias. To account for nonresponse and reduce the bias resulting from missing data, we must implement nonresponse compensation procedures and use analysis weights that account for survey nonparticipation. A design effect (DEFF) captures the variance inflation resulting from variation in weights from nonresponse adjustments. It is reasonable to assume a small design effect: DEFF = 1.10.

• The characteristic being measured is quantified as a population proportion of 50 percent.

Table B-3 presents results of the power analysis and the resulting precision level based on the above assumptions.

It is important to note that even though no sampling is involved in selecting the CCAMPIS grantee group of institutions (all 352 grantees from 2001 and 2002 are included in the study), the table presents calculated standard errors based on the assumption that nonresponse exists.⁶ In Table B-3, we treat the respondents within each group as a random sample from the 352 institutions and consider the number of institutions responding to the survey as the sample size for computing the standard errors. Furthermore, nonresponse adjustments made through weighting will result in a DEFF larger than 1 owing to the unequal weights resulting from nonresponse adjustment.⁷

Table B-3 shows that the minimum detectable difference (MDD) is a measure of the smallest difference between the CCAMPIS and non-CCAMPIS institutions that the study design is able to detect with 80 percent power and at a 90 percent confidence level. For example, an MDD equal to 11 percentage points means that, if 50 percent of the non-CCAMPIS grantee institutions have any contracts with community child care providers, then at least 61 percent of the CCAMPIS grantee institutions must have contracts with community child care providers in order for analysis to detect a statistically significant difference between CCAMPIS and non-CCAMPIS institutions, based on the 299 responding institutions (85 percent) in each group.

TABLE B-3

SAMPLE SIZES AND MDD BETWEEN CCAMPIS GRANTEE AND SIMILAR NONGRANTEE INSTITUTIONS

Sample	Initial Sample Size	Total Response Rate (percent)	Target Number of Completes	Approximate Design Effecta	Standard Error (percent)	Coefficient of Variation (percent)	Margin of Error (percent)b	MDD at 80% Power and 90% Confidence for Comparisons
Within-Group Descriptive Analyses
Grantees	352	85	299	1.1	3.03	6.06	5.00
Nongrantees	352	85	299	1.1	3.03	6.06	5.00
75 percent subgroup of institutions		85	244	1.1	3.50	7.00	5.78
50 percent subgroup of institutions		85	150	1.1	4.29	8.57	7.07
Comparison Grantees versus Nongrantees
Full sample of institutions		85	299	1.1				10.62
75 percent subgroup of institutions		85	244	1.1				12.24
50 percent subgroup of institutions		85	150	1.1				14.94
Total Sample	704	85%	598

^a A design effect of 1.1 is used to account for an increase in standard error due to the weighting adjustment for nonresponse. The sample size estimation used in the table was the overall or 100 percent, 75 percent, and 50 percent domains of population.

^b Margin of error (i.e., the half-width of the 90 percent confidence interval) for a proportion (p) near 0.50 is based on the binomial distribution. The sampling variance is projected in accordance with the model Var(p)=p*(1-p)/n. The margin of error = 1.65*square root[Var(p)]. The MDD for a one-sided test of
p1-p2 = 0 with alpha = 0.10 and power of 80% is MDD = SQRT{DEFF[Var(p1)/n + Var(p2)/n]}* ((z(alpha) + z(Beta)), where z(alpha) = 1.65 and
z(beta) = 0.84.

The computation of the prospective MDD in Table B-3 was based on an assumption that the proportion of institutions with the characteristic being assessed equals 50 percent, which yields a conservative estimate of standard error and hence a conservative MDD. For characteristics with proportions other than 50 percent, the MDDs may be smaller. Table B-4 presents the magnitudes of MDDs for different combinations of CCAMPIS population proportions computed with a 90-percent confidence level, 80-percent power, sample of 299 responders in both groups, and DEFF = 1 (assuming no variability in the weights).

TABLE B-4

POPULATION PROPORTION IN CCAMPIS GROUP, MDD, AND POPULATION PROPORTION
IN THE COMPARISON GROUP

p1 (CCAMPIS)	MDD	p2 (Comparison Group)
50	10.17	60.17
55	10.12	65.12
60	9.96	69.96
65	9.70	74.70
70	9.32	79.32
75	8.81	83.81
80	8.13	88.13
85	7.26	92.26
90	6.10	96.10

Estimation and Variance Computation. The data in our analysis will be weighted to account for institution nonresponse. We will create a weight for each institution to be computed by using a standard weighting class method or a response propensity modeling method (Kalton and Maligalig 1991; Holt and Smith 1979; Oh and Scheuren 1983; Vartivarian and Little 2003).

Along with the weighted survey estimates, we will compute the standard errors of the estimates. Variance/standard error estimation will take into account the weighting adjustment process as well as the assumption that respondents are a random sample of the CCAMPIS/non-CCAMPIS population.

3. Methods To Maximize Response Rates

Web-based data collection will help maximize response rates by allowing respondents to complete the survey at their convenience. Further, the survey’s integrated skips and automation features will allow respondents to move seamlessly from question to question without spending time reading and interpreting skip instructions, as is required on a standard mail survey. In addition, the Web-based survey will have a “save” option that permits respondents to start the survey and then complete it at a later time, minimizing the chance of mid-survey break-offs.

Not all respondents will have Internet access, and some with access may be uncomfortable responding to a Web-based survey. To maximize participation from these individuals and reduce nonresponse bias that may result from their nonparticipation, we will offer opportunities to complete a standard mail survey or telephone survey.⁸

We will use standard techniques to reduce nonresponse by providing evidence of legitimacy in an advance letter, FAQs, and reminder prompts via emails, letters, and telephone calls as appropriate. We will also offer a project-specific MPR email address and toll-free telephone number so that participants with questions or concerns about participation may contact us.

Despite our best efforts at minimizing nonresponse, some institutions (both CCAMPIS and non-CCAMPIS grantees) will inevitably fail to participate. We have planned a statistical approach to deal with nonresponse as described below. The adjustment process will implement a standard weighting class or response propensity model method.

4. Tests of Procedures or Methods

After MPR and ED thoroughly tested all aspects of the Web-based survey, MPR pretested the survey with one respondent at each of nine institutions (including both CCAMPIS grantees and nongrantees) during summer 2006. Because of pretest respondents’ difficulties completing some items, we proposed modifications to facilitate respondents being able to obtain Pell Grant data. We tested those modifications with 10 percent of the sample in a Phase I data collection from January to March 2007. Specifically, we examined issues of data availability, data quality, and respondent burden in Phase I. We note two major findings here (detailed results can be found in Appendix G):

1. Pell Grant Data. Most respondents lacked data to answer Pell Grant data questions. Tools provided to child care directors to facilitate their requesting these data from other institutional offices failed to achieve their goal: The other institutional offices were often unwilling or unable to provide the data. When Pell Grant data were provided (by either child care directors or another office), respondent burden was excessive. Given the low item response rates for these items, the data could not be interpreted with confidence.

2. Off-Campus Center Data. Most child care directors were unable to provide data on off-campus centers’ operations and services or the number of their institution’s students using such centers.

Accordingly, we eliminated almost all questions on Pell Grant data and off-campus centers.⁹ A few items with higher response rates were retained, and items on children of Pell Grant students were modified to ask about children of postsecondary students in general who use the institution’s child care services. A table identifying items that were retained, modified, or dropped can be found in Appendix A. The reduced burden should help us surpass the 77 percent response rate obtained in Phase I.

5. Individuals Consulted on and Responsible for Statistical Design

Amang Sukasih

Mathematica Policy Research, Inc.

Washington, DC

202.484.3286

Sameena Salvucci

Mathematica Policy Research, Inc.

Washington, DC

202.484.4215

Jill Constantine

Mathematica Policy Research, Inc.

Washington, DC

609.716.4391

1 The sample contains 228 institutions from the 2001 cohort and 124 institutions from the 2002 cohort, rather than the 222 and 122 grantees reported in Table B-1, since three CCAMPIS grants were awarded to community college districts that encompassed from three to five individual institutions. In this study, each community college district grantee is represented by its individual institutions. We matched each institution covered by these grants (the matching process is described below) for the three community college districts; each will be asked to complete a survey.

2 Selection bias refers to differences between the two groups (in this case, CCAMPIS and non–CCAMPIS institutions) due to unobserved covariates.

3 Postsecondary institutions contained in the sample frame were imported into ArcMap 9.1 and mapped using latitude and longitude data provided by the National Center for Education Statistics (NCES). Buffer rings were drawn around each institution. The radius of each buffer ring was based on NCES urban-centric codes and on estimates of reasonable distances for a postsecondary student to travel for child care, given the institution’s degree of urbanicity. Institutions in cities have a buffer ring with a 5-mile radius, in suburban areas a 10-mile radius, in distant town and rural areas a 20-mile radius, and in remote towns and rural areas a 35-mile radius. Then, block group data (2006 Census-based estimates from Geolytics) within each buffer area were aggregated and appended to the institutional record. Available data included the percentage of population by race and age group, percentage of households by income group, percentage of owner-occupied households, and percentage of renter-/other-occupied households.

4 Caliper is defined as the maximum absolute difference between the value of propensity score of the grantee institution and that of the non-grantee institution allowable in the matching.

5 For dichotomous or categorical variables, summary statistics can be computed as proportions. For continuous variables, summary statistics can be computed as means or medians.

6 With the use of a census rather than a sample survey, no sampling/standard error is involved because no sampling takes place. In this case, an analysis usually compares outcomes directly across groups without performing statistical hypothesis testing.

7 A computation that assumed the design effect equals 1 (i.e., no weighting adjustment was made) resulted in an MDD of 10.13 percent based on a sample size of 299 respondents in both groups.

8 All surveys completed in Phase I were submitted via the Web.

9 We are examining the feasibility of using financial aid administrative records to obtain persistence and graduation rates of Pell Grant recipients with children at CCAMPIS and non-CCAMPIS institutions.

17