SSB_NSCAW Design Option_9-15-20_FINAL

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Survey of National Survey of Child and Adolescent Well-Being (NSCAW) Adopted Youth, Young Adults, Adults, and Adoptive Parents

OMB Information Collection Request

0970 – New Collection

Supporting Statement

Part B

September 2020

Submitted by:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, DC 20201

Project Officers:

Amanda Clincy Coleman, PhD

Molly Jones, MAT, MA

Part B

B1. Objectives

Study Objectives

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services seeks approval for a one-time study to examine familial outcomes after a child’s adoption from the child welfare system. The primary objectives of this study include the following:

1. To understand the extent to which families who have adopted children who have exited the foster care system experience instability¹.

2. To understand risk and protective factors associated with post adoption instability.

To meet these objectives, the study will conduct new surveys with adopted youth, young adults, and adults who were participants in the first or second cohort of the National Survey of Child and Adolescent Well-Being (NSCAW I, II). The study will also conduct new surveys with adoptive parents. Data will be used to increase the child welfare field’s knowledge of instability and inform possible future research activities and technical assistance around promoting permanency for children after adoption.

Generalizability of Results

This study is intended to present an internally valid description of the extent of instability, and risk factors for instability, among children adopted after living in foster care. It is not intended to promote statistical generalization to other populations of children, youth, and young adults.

Appropriateness of Study Design and Methods for Planned Uses

The study design selected is intended to maximize the inclusion of children who exited foster care into adoption and their adoptive parents, as these respondents are best suited to answer the research questions of interest. The study is designed to allow for a long post adoption period (8 years or more) to increase opportunities to observe post adoption instability. Although we know that instability may occur many years after adoption is finalized, prior empirical studies of post adoption or guardianship instability had relatively short follow-up periods². The current study also takes advantage of two rich longitudinal datasets, NSCAW I and NSCAW II. We will extract pre-adoption child and family characteristics from the NSCAW datasets and combine them with new study data to enrich and expand upon the existing NSCAW datasets. This survey sample is not intended be not be representative of all adopted youth, young adults, and adults who have exited foster care, only those adopted youth, young adults and adult who participated in the NSCAW I/II surveys. The survey sample will also not represent the full youth population. This context, and the limits to representativeness, will be clearly documented in the text of all written materials associated with this study.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and Design

Target Population

The Survey of NSCAW Adopted Youth, Young Adults, Adults, and Adoptive Parents will be a one-time information collection effort. The study will include 588 adopted youth, young adults, and adults (15-37 years of age) and 554 adoptive parents who were participants in one of the NSCAW I or II longitudinal study cohorts and meet eligibility requirements described below. The NSCAW I cohort includes 6,228 children, ages birth to 14 (at the time of sampling), who had contact with the child welfare system within a 15-month period that began in October 1999. The NSCAW II cohort includes 5,800 children, ages birth to 17.5 years (at the time of sampling), who had contact with the child welfare system beginning in 2007. In both NSCAW I and II, a current caregiver respondent (biological or adoptive parent, kinship caregiver, or foster parent) was identified for every child at each wave of data collection. All NSCAW I adoption cases will be over the age of 18 at the time of new data collection. Preliminary NSCAW II data analysis suggests that 25% of eligible NSCAW II adoption cases will be 15–17 years old.

Sampling

A subsample of NSCAW I and II participants with a history of living in foster care before adoption will be eligible to participate in this study. Youth, young adult, and adult participants will all have been identified as living with an adoptive parent prior to the last wave of NSCAW I or NSCAW II data collection (see Attachment C for NSCAW Sampling Data Elements). Eligible adoption cases will include those who:

• Have a history of foster care placement (prior to adoption),

• Achieved adoption status prior to the end of NSCAW I or II, and

• Are 15 years or older at the start of data collection.

An age criterion (15 years and older) was included in sample eligibility to ensure that all youth participants had experienced some portion of their adolescence. The literature suggests that adolescence is a period of high risk for instability events, regardless of the age of adoption finalization (White et al., 2018).

Certain cases will be excluded from the adoption sample:

• Child cases who had cognitive disabilities according to NSCAW I and II data collection efforts. To avoid interviewing adopted youth, young adults or adults with cognitive disabilities, we will identify child cases that had a standardized IQ score 2 or more standard deviation under the mean in cognitive scales, which correspond to a score < 70. Thus, we will only interview the adoptive parent of youth, young adults or adults who at the last available Kaufman Brief Intelligence Test and Woodcock Johnson Tests of Academic Achievement from NSCAW I or II show a standardized score < 70 in both tests. Similarly, if the caregiver reported in the last available interview that the child has mental retardation or Down’s Syndrome, the child will be excluded from the study. For child adoption cases with cognitive disabilities, we will only complete the adoptive parent interview.

• Cases where the adoptive parent or adopted youth during NSCAW I or II explicitly asked not to be re-contacted for future research.

Sample Selection

Eligible adoption cases from NSCAW I and II will be identified by a derived variable in both the NSCAW I and NSCAW II datasets. This variable (“ever adopted”) is based on caregiver report that the current caregiver is an adoptive mother or father. If that information is missing, the derived adoption variable includes the caregiver or child instrument general variables from the case initiation database. This includes information from data collected during the sampling process from the designated NSCAW study child welfare agency liaison, that was then confirmed during the caregiver or child interviews. In the case of NSCAW II, Adoption and Foster Care Analysis and Reporting System (AFCARS)³ files are available through Wave 2 to indicate that the child exited foster care to permanency through adoption.

The proposed study involves a finite rather than a random sampling frame—those cases who were adopted by the end of the NSCAW I and II study periods. Consequently, the maximum number of possible respondents will be set at the start of data collection; new sample will not be eligible for release if response rates are lower than expected. For that reason, we have purposefully selected what we believe are reasonable response rate targets to guide our estimated sample size assumptions. Our response rate expectations take into consideration the 13 years that have elapsed since the last contact with NSCAW I participants. We have not conducted sample maintenance activities with this group since 2007.

Precision to Estimate Key Outcomes

Sample size requirements were obtained using power analysis calculations. We computed the minimum detectable effect size (MDES) for the types of planned analyses, with a primary focus on estimating the occurrence of both formal and informal instability events. All power analyses were conducted using PASS 14 (2015) assuming 0.8 power and a Type I error rate of .

Prior research estimates that 5–20% of children and youth who leave foster care for adoption experience instability (Berry, Propp, & Martens, 2007; Hartinger-Saunders, Trouteaud, & Matos Johnson, 2014a; Rolock, 2015; Rolock & White, 2016; Testa, Snyder, Wu, Rolock, & Liao, 2015). This research shows higher instability estimates for groups of children who had high risk factors (e.g., older age, behavior difficulties). The project team anticipated that approximately 5–20% of the NSCAW adoptee sample will experience instability (either formal or informal). Since some research questions propose to examine formal and informal instability separately, the project team also considered power to examine differences in these types of instability outcomes. Based on the literature, the project team expects that formal instability will occur in approximately 10% of all adoption cases when followed 10 years post adoption. There is no direct empirically based estimate of informal instability upon which to base a power analysis. However, drawing from the literature, the project team assumes that informal instability will occur in approximately 5% of the NSCAW I/II adoptee sample.

For estimating the occurrence of either formal or informal instability, an acceptable margin of error may be achieved with a sample size of 250 adoption cases. To estimate the occurrence of instability in the adoptee group, with a sample size of 271, the size of the 95% confidence interval will be ±3% to ±4%. For models designed to examine risk or protective factors for instability among adoption cases, we should be able to detect medium effect sizes for predictors of formal instability with a sample size of 250 cases. However, for models designed to predict informal instability (which we expect may have a lower occurrence), the study will require a sample size more comparable to 500 cases to detect medium effects. Consequently, we have targeted a sample size of slightly more than 500 adoptee and adoptive parent cases.

B3. Design of Data Collection Instruments

Development of Data Collection Instrument(s)

The main objective of the data collection is to identify adopted youth and young adult’s experiences of instability between adoption and before 18 years old, child age at the time of instability events, main reasons for instability, and services received at the time of instability. Design of data collection instruments was guided by the following specific objectives:

1. Estimate the occurrence of formal (foster care) and informal (residential treatment, runaway, etc.) instability among children adopted from the child welfare system; and

2. Identify risk and protective factors associated with increased or decreased risk of formal and informal instability, respectively.

To streamline the creation of data collection instruments, the project team identified topic areas to which adopted youth, young adults, adults, and adoptive parents responded when sample members participated in NSCAW I or NSCAW II. For example, the NSCAW datasets have detailed information about the child’s and family of origin’s involvement with the child welfare system, services needed and received from the time of the maltreatment report, assessments of child and adoptive parent well-being, foster care placement history, living situation permanency, maltreatment characteristics, and child maltreatment recurrence. The current study will capitalize on existing secondary data to understand these constructs. The new adopted youth/young adult and adoptive parent surveys were designed to gather new information not collected during the existing waves of NSCAW I and NSCAW II data collection. The surveys do ask again about basic sociodemographic information; this information needs to be updated because most children who participated in NSCAW are now adults.

Two instruments were developed for this study, one for youth or young adults, and one for their adoptive parent.

The youth/young adult/adult survey was developed using instruments from previous national or state surveys. The main sources of questions were the National Survey of Child and Adolescent Well-Being (child instruments used with cohorts I to III) (OMB #0970-0202), the Survey of Youth Transitioning from Foster Care (OMB #0970-0546), the Texas Youth Permanency Study, the Midwest Evaluation of the Adult Functioning of Former Foster Youth, and the Quality Improvement Center for Adoption & Guardianship Support and Preservation State Surveys⁴. Sources used for survey question development are listed in Attachment D. Items from these measures were adapted to remind the respondent about the reporting period for events of interest for this study—after adoption and before the youth turned 18 years old. The project team used a few questions from the National Survey of Adoptive Parents and Beyond the Adoption Order (British study) that focused on adoptive parents. These questions were adapted to be used with youth and young adults. Instrument 1: Survey of Adopted Youth, Young Adults and Adults (SAY) provides the youth/young adult interview items for the survey instrument.

Similarly, the adoptive parent survey was developed using the National Survey of Child and Adolescent Well-Being (caregiver instruments used with cohorts I to III), the National Survey of Adoptive Parents, the Quality Improvement Center for Adoption & Guardianship Support and Preservation State Surveys for parents and guardians and, Beyond the Adoption Order study. Sources used for survey question development are listed in Attachment D. Items from these measures were adapted to remind the parent of the reporting period for events of interest for this study—after adoption and before the youth turned 18 years old. The project team used a few questions from the Survey of Youth Transitioning from Foster Care, which is administered to youth. These survey questions were adapted to be used with adoptive parents. Instrument 2: Survey of Adoptive Parents (SAP) provides the parent interview items for the survey instrument.

For both instruments, the project team also developed items specifically for this project. The instruments will be pretested with nine individuals to identify language comprehension issues. After pretesting, the instrument will be revised to improve comprehension and usability. Any changes will be submitted to OMB review and approval as a change request.

Both youth and adoptive parent surveys include short upfront sections with about 20 items for demographic information, current household, and adoption history. The main section of the survey, about 90 items, focuses on youth experience of instability both formal (foster care) and informal (living at other adults’ household without adoptive parents, residential treatment, group home, running away, and use of homeless shelters). Shorter sections on quality of family relationships, family functioning, protective factors, mental health, adoption motivations, and receipt of post adoption informal and formal support.

Because the Survey of NSCAW Adopted Youth, Young Adults, Adults and Adoptive Parents focuses on instability events for youth and young adults who were/are adopted from child welfare system, it deals with several private and sensitive topics. This information is necessary to address the study’s core research questions and is not reliably available from other sources. The survey includes potentially sensitive questions about, for example, homelessness and runaway attempts. Adoption status and involvement with the child welfare system are also sensitive issues. Since some adoptee may not be aware of their adoption status, all participants will be asked if they have ever been adopted. If the response is “No”, the instrument’s skip pattern will route them to questions that exclude any mention of adoption. The instrument also excludes the concept of “adoptive parents” and instead just refers to “parents.”

B4. Collection of Data and Quality Control

Collection of Data

The contractor, RTI International, will collect all data for this study. All surveys completed by adults over the age of 18 years will be conducted via the web or by telephone. Outreach to adopted adults will occur at the same time as outreach to adoptive parents. The completion and timing of one interview does not impact completion or timing of the other interview. In other words, the adopted adult’s interview does not need to be completed prior to their adoptive parent’s interview (or vice versa).

Youth respondents 15–17 years will only be offered the opportunity to complete the survey by telephone. In this case, the adoptive parents of adopted youth 15-17 years will always be contacted first to acquire parental consent. Outreach to adopted youth will only occur after consent has been granted by the adoptive parent.

Recruitment Protocol

Using the updated information from tracing and locating efforts (described in Section B5), the project team will send all young adult/adult and adoptive parent sample members an advance mailing, or lead letter (Attachment E) as well as Fact Sheets, announcing the launch of data collection and providing information about the study. The letter will provide information on the background of the research effort, sponsorship, and contact information for the project team should respondents have questions about participation or prefer to complete the survey by telephone. The letter will also include instructions for accessing the web version of the survey for adoptive parents and young adults. For youth respondents under the age of 18 years, the letter will be mailed only to the adoptive parent because consent from the adoptive parent will be required prior to contacting the youth.

After advance letters have been mailed, we will make telephone contact attempts at different times of day and days of week until we reach the respondent. We will provide a dedicated toll-free hotline for support to sample members who have questions about the research or want to verify the legitimacy of the survey. Calls received via the toll-free hotline will route automatically to interviewers who are available and trained on the project. The contractor’s weekday, weeknight, and weekend staff coverage will allow sample members to quickly reach a person familiar with the study. Telephone interviewers will also provide the toll-free number when they leave voicemail messages during call attempts. If respondents are reached during in-person field locating efforts (see Section B5), the field representative will immediately show the respondent how to access the web survey and offer a telephone number for where to call to complete the telephone survey.

Study materials designed to be distributed after survey completion (e.g., resource lists) are included in Attachment I).

Informed Consent Procedures

Adults over the age of 18 years will be asked to provide their consent prior to study participation. A telephone interviewer will read the consent form to adults verbatim, to remind them about how they were selected, to emphasize the privacy of the data collected, and to make sure they understand that participation is voluntary and that they have the right refuse to answer any questions in the survey. Web survey participants will read this same consent form information online and will provide electronic consent prior to completing the web survey (Attachment G, Consent Form for Young Adult, Adult and Adoptive Parent Respondents).

A copy of the consent form will be included with the Lead Letter to Parents and Adults. Participants who complete the web survey will also be able to print a copy of the consent from the website. If requested by the participant completing a telephone interview, a copy of the consent form will be mailed/emailed to them again.

Parental Consent and Youth Assent Procedures

For youth 15–17 years old, we will contact and interview the parent first. If the parent completes the web survey, the telephone interviewers will call the parent as soon as the web survey is received by RTI. The telephone interviewers will use the Parent Permission for Child Participation form (Attachment H) to describe the interview with the child. Telephone interviewers will remind parents about the study’s purpose, procedures, and other key information. The telephone interviewers will also stress the importance of providing the youth with privacy during the completion of the survey. As the instruments are very similar, the telephone interviewers will be trained to explain that most of the questions will also be asked to youth, but that if they respond that they have never been adopted, the questions will not include any reference to their adoption status. This information is also included in the fact sheet (Attachment E) and the phone and web versions of the parent consent form (Attachment H). If the parent has questions to which the telephone interviewers can’t respond and/or the parent requests to talk with the Project Director (PD) or IRB representative, the telephone interviewers will provide the toll free number for the PD and IRB. The telephone interviewers will complete an incident report and alert the PD within 24 hours that a parent is requesting a call.

Upon securing verbal parental permission by phone, the project team will acquire the adolescents’ verbal assent. If the parent does not agree to provide permission for the child’s participation, the telephone interviewers will thank them, and the study team will send the parent the “Caregiver Refusal Minor Child-Refuses” letter (Attachment E) requesting to interview the child. After a week, the telephone interviewers will make one final attempt calling the parent to ask if they received the letter and have reconsidered their decision, if that is the case, they will request permission for the child interview.

If the parent provides permission for the child minor interview, we will attempt to complete the minor interview when the minor is at the consenting parent’s home. Before launching the survey, interviewers will describe the need for a private setting to the youth and options to consider if a private setting cannot be achieved. For example, the interviewer may offer to schedule the interview at another time when a private setting can be established.

Data Collection Procedures

For adults, the surveys will be conducted using the web or by telephone, at the choice of the respondent. All youth will complete the survey by telephone. Because survey respondents often have concerns about the length of a telephone survey, telephone interviewers will provide options for respondents to consider, including the web version of the survey. Telephone interviewers will offer to complete the survey across multiple phone calls if necessary. Similarly, respondents can start the web survey, save their work, and complete it at another time. For sample members expressing concerns about privacy or a preference to complete the survey online, telephone interviewers will provide unique login credentials for the web survey. The survey will take an average of 30 minutes to complete. As noted in Supporting Statement A, youth/young adult interview modules and items are provided in Instrument 1: Survey of Adopted Youth, Young Adults and Adults (SAY) and the adoptive parent interview modules and items are provided in Instrument 2: Survey of Adoptive Parents (SAP).

While taking part in the study will not present physical risks to parent and their child, some questions might make respondents feel uneasy (e.g. questions from the instability section and the factors that triggered household’s instability). For phone interviews, a distressed respondent protocol will be available. Telephone interviewers will receive specific training on how to handle respondents who become upset by a question or a series of questions and how to offer the appropriate support. Support includes offering breaks, allowing respondents to refuse questions they are uncomfortable answering, and connecting them to professional assistance when needed.  As part of training, the study team will also emphasize the need to avoid disclosing adoption status. Across the training, interviewers will be reminded at several sections of the training (overall study description, consent and assent process, youth instrument) that if the child (former NSCAW index child) states that they are not adopted, then the interviewer will not reveal to them that they are adopted.

Data Quality

Web and telephone survey platforms improve survey data quality by eliminating routing errors and implementing logical range checks. The quality of data gathered in this study will also depend upon the abilities of the field data collection staff. These staff will be responsible for tracing and locating efforts as well as in-person participant recruitment. Field tracing staff will receive training on study locating and sample recruitment protocols. Ensuring telephone interview quality will require that telephone interviewers receive training on study procedures and survey administration. During training, each interviewer’s performance will be evaluated, and additional training provided as necessary to ensure that each has the skills required for the study. The final component of training will involve certification of interviewers in key areas of performance, including answering questions about the study, gaining cooperation, administering consent forms, and administering the survey.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

NSCAW I and NSCAW II are longitudinal data collection efforts. To facilitate longitudinal data collection in future waves, participants were asked to provide their current address, telephone number, 2-3 secondary points of contact (and their addresses/phone numbers), driver’s license number (if 16 years or older), and the last 4 digits of the NSCAW child case’s social security number. NSCAW I and II include surveys with both children/youth and parents or caregivers. Young adult surveys do not have a corresponding parent survey but would have had a parent survey in a prior wave. In our experience, young adult respondents also often list their parents as a secondary point of contact. Consequently, the contractor has locating information on selected youth/young adult respondents as well as adoptive parent respondents.

For eligible NSCAW I youth/young adult/adult and adoptive parent participants, we anticipate a 60% response rate. Sample maintenance activities have not been conducted with this group since 2007. As a result, the contractor has planned that 100% of eligible and sampled NSCAW I respondents will require intensive tracing and locating activities (i.e., calling secondary contacts in addition to mail and computerized locating searches). The project team has also assumed that 75% of youth/young adult respondents will require in-person field-based locating efforts. A contact card (Attachment F) has been designed to be mailed as part of the locating effort outreach. These tracing and locating plans are designed to attain the expected response rate. Since it has been 13 years since the contractor’s last contact with NSCAW I participants, the project team does not expect to exceed a 60% response rate.

For eligible NSCAW II youth/young adult/adult and adoptive parent participants, we anticipate a 68% response rate. Sample maintenance activities have not been conducted with the NSCAW II sample since 2012. Since the NSCAW II sample was contacted more recently (~8 years ago) than the NSCAW I sample, fewer tracing resources and attempts to locate each NSCAW II sample member will be required. The project team has still assumed that 100% of eligible and sampled NSCAW II cases will require intensive tracing and locating efforts (i.e., calling secondary contacts in addition to mail and computerized locating searches); however, we expect that only 60% will require in-person field-based locating efforts to achieve the expected response rate.

The project team based these response rate assumptions on our experience conducting the NSCAW I and II longitudinal surveys, as well as our experience collecting longitudinal data for the Add Health sample⁵. NSCAW II achieved youth/young adult response rates of over 80% in Wave 2 (18-month follow-up) and Wave 3 (3-year follow-up); importantly, a subset of youth were over the age of 18 years (Casanueva, Tueller, Smith, Dolan, & Ringeisen, 2014; Casanueva et al., 2012). The contractor achieved a 53% response rate for a web/mail survey during Wave 5 of the Add Health survey. Add Health Wave 5 data collection was conducted 8 years after Wave 4 with adults who at the time (of Wave 5) were 31–42 years old. The Add Health survey did not include in-person field-based locating efforts and for that reason, we have estimated a slightly higher response rate for the Surveys of NSCAW Adopted Youth, Young Adults, Adults, and Adoptive Parents.

A completed interview from both the adopted young adult/adult and adoptive parent will not be necessary to analyze the key instability outcomes of interest.

Many study findings will be interpretable even if the study yields a <60% response rate. Prior findings around post adoption instability are weakened by sample sizes even smaller than what this study would achieve without a 60% response rate. Prior studies are also weakened by short follow up periods within which to observe instability outcomes after adoption. With the secondary data available from NSCAW I/II along with the length of time passed since adoption, the PAGI study is well positioned to advance this field of knowledge even with a lower than expected response rate.

To further consider the impact of a lower-than-expected response rate, our team revisited the power analyses conducted for this study and discussed in Section 2B (page 4). A 60% response rate will result in a sample size of approximately 438 cases. For estimating the frequency of either formal or informal instability (RQ1), we can achieve an acceptable margin of error with a sample size of 250 adoption cases. For predictive models designed to examine risk protective factors for any instability (summing across types of instability) among adoption cases (RQ 2), we will be able to detect medium effect sizes with a sample size of 250 cases. However, a sample size closer to 500 will be needed to detect medium effect sizes for predictive models of specific types of instability (e.g., formal versus informal). So, the analytic goals hampered by a lower-than-expected response rate (and smaller resulting sample size) are those more narrowly focused research questions designed to specifically understand the lowest frequency forms of instability outcomes.

Nonresponse

Nonresponse bias analyses will be conducted to understand the representation of those who completed the survey of adopted youth, young adults, adults, and adoptive parents compared to those who did not. The project team will compare responders to nonresponders using secondary data about child and family characteristics from the NSCAW I/II baseline data collection. Adopted youth and adults with a history of instability may be some of the most difficult to locate and interview. Secondary data will include several characteristics associated with heighted risk for post adoption instability (e.g., older age, prior relationship with adoptive parent, number of out-of-home placement changes prior to adoption). The NSCAW I/II data also includes information about instability that occurred after adoption but during the original NSCAW I/II study period. We will also compare responders to nonresponders using this NSCAW I/II secondary data about observed post adoption instability events.

The project team will use statistical comparisons to identify variables associated with nonresponse. If some variables are associated with nonresponse, the project team will make nonresponse bias adjustments using poststratification nonresponse weighting. Using weight tests (Bollen, Biemer, Karr, Tueller, & Berzofsky, 2016), we will assess the bias/variance tradeoff. Specifically, we will test whether the nonresponse weights yield a significant reduction in bias relative to the increase in variance caused by the weights. Specifically, weights can increase standard errors with reduced statistical power. Hence, weights should not be applied unless the bias reduction is nontrivial.

B6. Production of Estimates and Projections

Estimates produced by this work will be prepared for internal use by ACF and external release by the agency in the form of presentations, reports, and/or publications. These publications will include information that is not generalizable to a nationally representative sample of all adopted children and youth. We will make it clear within these publications that this information is not generalizable to all adopted children and families or to the general youth population. Consequently, the data will not be used to generate overarching national population estimates, either for internal use or dissemination. However, the representativeness of the adoption sample who completed surveys for this study will be compared to the characteristics of all NSCAW I and II adopted cases (from which the sample was drawn).

B7. Data Handling and Analysis

Data Handling

This survey will be programmed for web and telephone administration. This technology affords several improvements in the collection of survey data specific to mitigating and correcting detectable errors and minimizing the errors typically produced by data entry by hand. First, this technology improves the consistency of data that a respondent provides. All response categories will be given an acceptable, fixed range. A respondent to the web survey (or the telephone interviewer) will not be able to enter a response that falls outside the logical range. This reduces the need for subsequent data editing. Second, the web/telephone survey technology provides greater expediency for data processing and analysis. Several backend processing steps, including editing, coding, and data entry, become a part of the data collection process and the survey responses load directly into the final data file. And, finally, the web and telephone survey programming platforms allow the order of response categories to be randomized across respondents. This helps eliminate the recency bias whereby respondents may be more likely to endorse those response options presented first to them for each survey question.

Data Analysis

The Survey of NSCAW Adopted Youth, Young Adults, Adults and their Adoptive Parents is poised to answer eight research questions (see Table 1). Prior to data analysis, the RTI team will derive instability variables. These variables will combine case-specific information from existing NSCAW I and II data with instability outcomes collected through the new surveys. We will carefully review for duplicative reports so as not to overestimate the occurrence of instability events (e.g., a young adult reporting a runaway attempt in adolescence that was already captured by that same youth’s report during the NSCAW study period). The project team will derive variables for each type of instability event separately—formal instability/foster care reentry, informal instability (homelessness, runaway, parental lockouts), periods living with other adults/relatives, and other temporary interruptions of care (residential treatment center, group home, juvenile detention). We will also provide descriptive information about child age at the first time of each type of instability event.

Along with the list of core research questions, Table 1 provides information on the analysis strategies likely to be used to address each research question. These analytic strategies are also detailed in the following section.

Several analysis strategies are planned for the Survey of NSCAW Adopted Youth, Young Adults, Adults, and Adoptive Parents. First, the project team will develop estimates to describe adopted youth/young adults/adults and adoptive parents who report childhood formal and informal instability events that occurred after adoption (RQ 1). Prior to data analysis, the RTI team will derive instability variables. These variables will combine case-specific information from existing NSCAW I and II data with instability outcomes collected through the new survey. The project team will derive separate variables for each type of instability event—formal instability/foster care reentry, informal instability (homelessness, runaway), periods living with other adults/relatives, and other temporary interruptions of care (residential treatment center, group home, juvenile detention). The project team will use univariate methods to derive estimates (proportions, their standard errors, and their margins of error) of important variables related to different types of instability events.

Second, the project team will explore and describe risk and protective factors for formal and informal instability after adoption through group comparisons and bivariate correlations (RQ 2). These include contingency table (crosstab) analyses with appropriate statistical tests (e.g., Pearson’s χ²) and logistic regressions. In logistic regressions, the instability indicators will serve as outcome variables, and risk and protective factors will be predictor variables. First, risk and protective factors for collinearity will be assessed. Both unadjusted and adjusted risk and protective factor effects will be estimated. The coefficients from the logistic regression will be transformed to odds ratios (ORs) to demonstrate unadjusted and adjusted effect sizes for the risk and protective factors. The confidence intervals for these ORs will also be computed. This will enable the project team to examine risk and protective factors for childhood formal and informal instability after adoption across age groups, genders, and other sociodemographic characteristics.

Table 1. Surveys of Adopted Youth/Young Adult/Adult and Adoptive Parents Research Questions and Analytic Strategies

Research Question		Analytic Strategy
RQ 1	To what extent have adopted NSCAW participants experienced post adoption formal and informal instability?	Proportion
RQ 2	What are the risk and protective factors for post adoption instability at the youth, adoptive parent, and family levels?	Crosstabulation, bivariate correlations, logistic regression
RQ 3	How are youth/young adult, adult, and adoptive parent self-reported motivations to sustain the adoption relationship associated with post adoption instability outcomes?	Logistic regression
RQ 4	How are young adult and parent self-reported expectations or perceptions of their adoption relationship associated with both formal (e.g., foster care reentry) and informal (e.g., periods of homelessness, runaway events) post adoption instability?	Logistic regression
RQ 5	How is the association between youth/adoptive parent/family characteristics and instability outcomes impacted (or moderated) by youth/young adult and adoptive parent self-reported expectations, motivations, or perceptions of their adoption relationship?	Logistic regression
RQ 6	What is the quality of current adoptive parent-child relationships among children who exited foster care to permanency through adoption?	Proportions
RQ 7	What support services such as peer support groups, individual or family counseling, or academic tutoring are accessible to youth/young adults and adoptive parents experiencing (or at risk of experiencing) post adoption instability?	Proportions, Crosstabulation
RQ 8	What are the facilitators and barriers to accessing support services for adoptive families?	Proportions

Similar logistic regression models will be used to assess the influence of adolescent/young adult and adoptive parent self-reported motivations to sustain the adoption relationship associated with post adoption instability outcomes (RQ 3). The project team will use logistic regression models to determine the degree to which the association between child/caregiver/family characteristics and instability outcomes can be impacted (or moderated) by young adult and adoptive parent self-reported expectations, motivations, or perceptions of their adoption relationship (RQ 4 and RQ 5). For example, these models will test whether an observed positive association between child age at the time of adoption and post adoption instability is weakened by a youth/young adult/adult or adoptive parent’s report of parental love and commitment.

Descriptive statistics (proportions and standard errors) will describe the self-reported quality of current and past adoptive parent-youth/young adult relationships among children who exited foster care to permanency through adoption (RQ 6). Through descriptive statistics (percentages and standard errors), the project team will summarize the frequency and proportion of types of self-reported support service use among adopted youth/young adults and adoptive parents, as well as perceived barriers and facilitators to service use (RQ 7 and RQ 8).

The special requirements of the Survey of NSCAW Adopted Youth, Young Adults, Adults, and Adoptive Parents may preclude using “off-the-shelf” methods in many circumstances and may require careful programming with sophisticated statistical software such as SUDAAN, MPlus, or R. The contractor has extensive experience with such programming and has applied these methods to the NSCAW data and in other similar research projects.

Missing Data

As with most studies, the Survey of NSCAW Adopted Youth, Young Adults, Adults, and Adoptive Parents will likely contain missing data because of variable nonresponse. As noted in Section B5, variable-specific nonresponse is expected to be small.

In prior work, the contractor found that NCSAW analyses with nonresponse can be addressed using methods for missing data, such as full information maximum likelihood (ML) estimation of regression curve models. Under conditions where full information ML was computationally infeasible, we have used multiple imputation, which approximates the ML solution when the number of imputed datasets is large. Both approaches have been shown to increase statistical power and reduce bias under most conditions.

None of these analyses will use the NSCAW I or II weights because this data is being generated for a truly different sample. Cases will be combined from NSCAW I and II for these analyses, so a variable will be created to mark cohort of origin for each case. This variable will be used as a control variable in analyses to help account for any cross-cutting cohort differences. However, if it is decided that weights for nationally representative estimates should be derived, those weights could be used with the logistic regression analyses if a weight test determines they significantly reduce bias (Bollen, Biemer, Karr, Tueller, & Berzofsky, 2016).

Data Use

ACF and RTI are committed to transparency in research. Sharing access to survey datasets will support further analyses that can inform child welfare practice related to post adoption instability. We will submit the final, deidentified survey data to the National Data Archive on Child Abuse and Neglect (NDACAN), following their requirements for ensuring privacy. The contractor will prepare supporting materials to contextualize and assist in interpretation of the data and as required by this Post Adoption Instability project dataset, including documentation of sampling methods, response rates, population of inference, construction of analytic variables, and construction and appropriate use of survey weights, along with a variable list and codebook.

Few datasets are available to examine the long-term outcomes of adopted children and their families. This dataset will allow analysts from the field not only to conduct their own analyses of post adoption instability, but also to analyze the health, family functioning, and service supports described by adopted youth and young adults as well as their adoptive parents.

B8. Contact Person(s)

Amanda Clincy Coleman

Senior Social Science Research Analyst

Office of Planning, Research, and Evaluation

Administration for Children and Families

U. S. Department of Health and Human Services

330 C Street SW, 4^th Floor

Washington, DC 20201

amanda.coleman@acf.hhs.gov | (202) 690-5976 

Heather Ringeisen

Director, Center for Behavioral Health and Development

Survey Research Division

RTI International

Post Office Box 12194

Research Triangle Park, NC 27709

hringeisen@rti.org | 919.541.6931

Attachments

Instrument 1: Survey of Adopted Youth, Young Adults and Adults (SAY)

Instrument 2: Survey of Adoptive Parents (SAP)

Attachment A: PAGI Project Conceptual Framework

Attachment B: Public Comments

Attachment C: NSCAW Sampling Data Elements

Attachment D: List of Sources used for Survey Question Development

Attachment E: Fact Sheets, Lead Letters and Refusal Letters (mail/email) for Youth/Young Adult/Adult and Parent Respondents

Attachment F: Contact Card

Attachment G. Consent Forms for Adult and Adoptive Parent Respondents

Attachment H: Assent Form for Youth and Parent Consent Form for Youth

Attachment I: Resource Guides for Respondents

References

Akin, B. (2011). Predictors of foster care exits to permanency: A competing risks analysis of reunification, guardianship, and adoption. Children and Youth Services Review, 33, 999-1011. http://dx.doi.org/10.1016/j.childyouth.2011.01.008

Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3), 399-424. http://dx.doi.org/http://doi.org/10.1080/00273171.2011.568786

Benkeser, D., Cai, W., & van der Laan, M. (2019). A nonparametric super-efficient estimator of the average treatment effect (Vol. arXiv:1901.05056v1 [stat.ME]): Cornell University.

Bollen, K.A., Piemer, P.P., Karr, A.F., Tueller, S., & Berzofsky, M.E. (2016). Are survey weights needed? A REview of diagnostic tests in regression analysis. Annual Review of Statistics and Its Application, 3: 375-392.

Carnegie, N. B., Harada, M., & Hill, J. (2018). treatSens: A package to assess sensitivity of casual analyses to unmeasured confounding (Version R package version 2.1.3). New York, NY. Retrieved from https://CRAN.R-project.org/package=treatSens

Corman, D. L., & Coon, L. S. (2007). DCFS pilot program: Permanency and stability for children in the care of elderly/frail adoptive parents and subsidized guardians (interim report to the majority leader, Illinois House of Representatives). Chicago, IL: Dept. of Children and Family Services, Center for Law and Social Work.

Rolock, N., & White, K. R. (2016). Post-permanency discontinuity: A longitudinal examination of outcomes for foster youth after adoption or guardianship. Children and Youth Services Review, 70, 419-427. http://dx.doi.org/10.1016/j.childyouth.2016.10.025

Rolock, N., & White, K. R. (2017, Oct). Continuity for children after guardianship versus adoption with kin: Approximating the right counterfactual. Child Abuse Negl, 72, 32-44. http://dx.doi.org/10.1016/j.chiabu.2017.07.001

Rolock, N., White, K. R., Ocasio, K., Zhang, L., MacKenzie, M. J., & Fong, R. (2019). A comparison of post-adoption placement stability for former foster youth in two large U.S. states. Research on Social Work Practice. http://dx.doi.org/10.1177/1049731518783857

Rosenbaum, P. R. (1987). Model-based direct adjustment. Journal of The American Statistical Association, 82, 387-394. http://dx.doi.org/http://dx.doi.org/10.1080/01621459.1987.10478441

Shortreed, S. M., & Ertefaie, A. (2017). Outcome-adaptive lasso: variable selection for causal inference. Biometrics, 73, 1111-1122. http://dx.doi.org/http://doi.org/10.1111/biom.12679

Testa, M. F. (2004, Winter). When children cannot return home: Adoption and guardianship. Future of Children, 14(1), 114-129.

Testa, M. F. (2010). Evaluation of child welfare interventions. In M. F. Testa & J. Poertner (Eds.), Fostering accountability: Using evidence to guide and improve child welfare policy (pp. 195-230). New York: Oxford.

White, K. R. (2016). Placement discontinuity for older children and adolescents who exit foster care through adoption or guardianship: a systematic review. Child and Adolescent Social Work Journal, 33(4), 377-394. http://dx.doi.org/http://dx.doi.org/10.1007/s10560-015-0425-1

 White, K. R., Rolock, N., Testa, M., Ringeisen, H., Childs, S., Johnson, S., & Diamant-Wilson, R. (2018). Understanding post adoption and guardianship instability for children and youth who enter foster care. Washington, DC: Office of Planning, Research, and Evaluation, the Administration for Children and Families, U.S. Department of Health and Human Services.

1 Instability is defined as including formal events, such as a child’s reentry into the foster care system, or informal events, such as a child going to live with a grandparent or running away.

2 For reference, NSCAW II followed-up with respondents in two waves after baseline at 18 months (Wave 1) and 3 years (Wave 2) after baseline.

3 AFCARS (OMB #0970-0422) “collects case-level information from state and tribal title IV-E agencies on all children in foster care and those who have been adopted with title IV-E agency involvement.” https://www.acf.hhs.gov/cb/research-data-technology/reporting-systems/afcars

4 These are not federal studies and so do not have an OMB number associated with them.

5 The National Longitudinal Study of Adolescent to Adult Health (Add Health) is a longitudinal study of a nationally representative sample of adolescents in grades 7-12 in the United States during the 1994-95 school year. The Add Health cohort has been followed into young adulthood with four in-home interviews, the most recent in 2008, when the sample was aged 24-32. Add Health re-interviewed cohort members in a Wave V follow-up from 2016-2018 to collect social, environmental, behavioral, and biological data with which to track the emergence of chronic disease as the cohort moves through their fourth decade of life.

12