Supporting Statement Part A 06-28-2023

NCEE System Clearance for

DESIGN AND FIELD STUDIES 2023-2026

OMB# 1850-0952 v.8

Supporting Statement Part A

Prepared by:

National Center for Education Evaluation (NCEE)

Institute of Education Sciences

U.S. Department of Education

Washington, DC

June 2023

TABLE OF CONTENTS

Part A JUSTIFICATION

A.1 Importance of Information 1

A.2 Purposes and Uses of the Data 3

A.3 Improved Information Technology (Reduction of Burden) 3

A.4 Efforts to Identify Duplication 3

A.5 Minimizing Burden for Small Institutions 4

A.6 Frequency of Data Collection 4

A.7 Special Circumstances 4

A.8 Consultations Outside the Agency 4

A.9 Payments or Gifts to Respondents 4

A.10 Assurance of Confidentiality 5

A.11 Justification for Sensitive Questions 5

A.12 Estimates of Burden 5

A.13 Total Annual Cost Burden 5

A.14 Annualized Cost to Federal Government 5

A.15 Program Changes or Adjustments 5

A.16 Plans for Tabulation and Publication 5

A.17 Display OMB Expiration Date 6

A.18 Exceptions to Certification Statement 6

1. JUSTIFICATION

1. Importance of Information

This is a request for a 3-year renewal of the generic clearance for the National Center for Education Evaluation (NCEE) that will allow it to collect preliminary or exploratory information to aid in study design. Specifically, the clearance will enable NCEE to: (1) field brief, quick turnaround surveys, extract test case administrative data, administer interviews, or conduct “mini-experiments” in advance of a study for the purpose of determining feasibility, a random assignment or comparison group strategy, or a data collection approach most suitable for a potential or planned evaluation (e.g., for a Federal Student Aid Experimental Site evaluation, a short collection to identify the conditions of an experiment in which an interested institution of higher education intends to participate); and (2) develop, test, and improve its survey and assessment instruments, methodologies, and study dissemination strategies.

The generic clearance is an important planning tool for NCEE.¹ Conducting these activities, and in particular conducting them earlier and more quickly, would enable NCEE to improve both the quality and turnaround time of its policy relevant evaluations, many of which are congressionally-mandated with ambitious dates for required reporting. The procedures expected to be used include but are not limited to exploratory surveys and interviews, focus groups, cognitive laboratory activities, pilot testing versions of an intervention or data collection approach, small-scale experiments that explore questionnaire design, incentives, or mode, and usability testing.

This request for generic clearance provides a description of the scope of possible activities that might be covered and the procedures for keeping OMB informed about them. NCEE requests the same conditions that have been included in the previous clearance agreement for Design and Field Studies (OMB# 1850-0952), approved on November 3, 2020. This system generic clearance request will go through the regular two Federal Register posting periods and OMB review and approval, subsequent to which, NCEE asks that OMB review and clear specific proposed generic information collection activities within a two-week period with no Federal Register Notice period required under the generic clearance. This is similar to NCES’s generic clearance process.

Examples of projects that have submitted design and field studies under this clearance in the last three years include the Federal Student Aid Experimental Sites Initiative Loan Counseling Experiment Cognitive and Usability Interview and the REL Usability and Feasibility Study for the Strategies to Improve Reading (STIR) Approach. We anticipate that other NCEE projects will also be able to use this clearance for developmental projects.

The methods proposed for use in NCEE study design and questionnaire and assessment development are as follows:

Design interviews. These may be conducted with individuals either in person, virtually, or via survey. The purpose would be to understand a topical area (such as a policy or program) or an intervention and may be used in the very early stages of developing an experiment or program evaluation and its accompanying data collection materials. They may cover discussions related to administrative records (e.g. what types of records are available, where, and in what format), subject matter, potential study sample, definitions, possibilities for random assignment, implementation approaches or challenges and so forth in order to assess study design and feasibility. Exploratory interviews may also be used in evaluating whether there are sufficient issues related to an existing data collection to consider a redesign.

Field test. For the purposes of this clearance, we are defining field tests as data collection efforts conducted among either purposive or statistically representative samples, for which the main objective is gathering critical information needed to inform the design of an experimental or program implementation evaluation. This may include collecting data as part of a pilot of an intervention, experimental study procedures, or data collection procedures. Field test or pilot collections are an essential component of this clearance package because they serve as the vehicle for ensuring that federal investments in education evaluations, including large-scale data collections, are feasible and appropriate and yield reliable and valid information. Under this clearance a variety of instruments may be used or tested as part of a pilot, and the exact nature of the surveys and the samples is undetermined at present. However, due to the smaller nature of the tests, we expect that some will not involve representative samples. In these cases, samples will basically be convenience samples, which may be limited to specific geographic locations or a subset of those potentially interested in participating in a larger-scale experimental evaluation. The samples may involve expired rotation groups of current survey blocks that are known to have specific aggregate demographic characteristics. The needs of the particular sample will vary based on the type of field test, including the respondent, content of the instruments, or data collection methods, but the selection of sample cases will not be arbitrary in any instance.

Behavior coding. This method involves applying a standardized coding scheme to the completion of an interview or questionnaire, either by a coder using a tape-recording of the interview or by a "live" observer at the time of the interview. The coding scheme is designed to identify situations that occur during the interview that reflect problems with the questionnaire. For example, if respondents frequently interrupt the interviewer before the question is completed, the question may be too long. If respondents frequently give inadequate answers, this suggests there are other problems with the question. Quantitative data derived from this type of standardized coding scheme can provide valuable information to identify problem areas in a questionnaire, and research has demonstrated that this is a more objective and reliable method of identifying problems than the traditional interviewer debriefing, which is typically the sole tool used to evaluate the results of a traditional field test (New Techniques for Pretesting Survey Questions by Cannell, Kalton, Oksenberg, Bischoping, and Fowler, 1989).

Interviewer debriefing. This method employs the knowledge of the employees who have the closest contact with the respondents. In conjunction with other methods, we plan to use this method in our field tests to collect information about how interviewers react to the survey instruments.

Respondent debriefing questionnaire. In this method, standardized debriefing questionnaires are administered to respondents who have participated in a field test. The debriefing form is administered at the end of the instruments being tested and contains questions that probe how respondents interpret survey questions or administrative data extraction requests and whether they have problems in completing the field test data collection.

Follow-up interviews (or reinterviews). This involves re-interviewing or re-assessing a sample of respondents after the completion of a survey or assessment. Responses given in the reinterview are compared with the respondents’ initial responses for consistency. In this way, reinterviews provide data for studies of test–re-test reliability and other measures of the quality of data collected. In turn, this information aids in the development of more reliable measures.

Cognitive and usability interviews. This method involves intensive, one-on-one interviews in which the respondent is typically asked to "think aloud" as he or she answers survey questions. A number of different techniques may be involved, including asking respondents to paraphrase questions, probing questions asked to determine how respondents came up with their answers, and so on. The objective is to identify problems of ambiguity or misunderstanding, or other difficulties respondents have answering questions. This is frequently the first stage in revising a questionnaire.

Focus groups. This method involves group sessions guided by a moderator, who follows a topical outline containing questions or topics focused on a particular issue, rather than adhering to a standardized questionnaire. Focus groups are useful for surfacing and exploring issues that are nuanced and conditional on factors not easily anticipated.

Procedures for Clearance

Before any activity is undertaken, NCEE will provide OMB with a generic information collection that includes a supporting statement describing the testing or exploratory data collection to be conducted and its intended purpose. The supporting statement will be similar to the “Volume I Supporting Statement” that NCES provides for each generic information collection under the NCES generic clearance. If relevant, a copy of instruments or debriefing materials will be included. Depending on the specific activity, this may be a printed questionnaire, a set of prototype items, or an interview script. When mini-experiments are conducted, either in small group sessions or as part of a field test, the different version instruments or interventions will be provided or described. NCEE requests that OMB raise comments on substantive issues within 10 working days of receipt.

Data collection for this project is authorized under the legislation authorizing NCEE evaluations. In most cases, this is the Education Sciences Reform Act (ESRA 2002, 20 U.S.C. §9563), though other legislation (e.g., HEA, 20 U.S.C. § 1094a) or Title 13 or 15 may apply for exploratory or pilot data collections conducted in concert with other offices within ED or other Federal agencies. If collections rely upon authority other than ESRA, the relevant authorizing statute will be specified.

2. Needs and Uses

The information collected under this program will be used by staff from NCEE and sponsoring agencies to design high-quality evaluations of federal education programs and improve the quality of surveys and assessments that inform them before those studies are conducted. None of the data collected under this clearance will be published for its own sake, though data collected through this clearance may be used during full-scale experiments and related data files and publications when appropriate. Data will not otherwise be made public, except when included in research reports prepared for sponsors inside and outside of NCEE. The results may also be prepared for presentations related to evaluation or survey methodology at professional meetings or publications on NCEE’s website and in professional journals.

Information quality is an integral part of the pre‑dissemination review by NCEE, which abides by NCES’s Statistical Standards and IES Style Guide (http://nces.ed.gov/statprog/standards.asp). Information quality is also integral to the information collections conducted by NCEE and is incorporated into the clearance process required by the Paperwork Reduction Act (PRA).

During the past three years, this generic clearance has been used for:

1. 1850-0952 v.1 Experimental Sites Initiative Loan Counseling Experiment Cognitive and Usability Interview

2. 1850-0952 v.2 NCER/NCSER Survey of Principal Investigators' Collection of Sample Demographics

3. 1850-0952 v.3 REL Midwest, Usability and Feasibility Study for the Strategies to Improve Reading (STIR) Approach

4. 1850-0952 v.4 Regional Educational Laboratory, Midwest Usability and Feasibility Study of the Supporting Inclusive and Diverse Educator Environments (SIDEE)

5. 1850-0952 v.5 Regional Educational Laboratory, Midwest Usability and Feasibility Study for the ENgagement and Achievement through Computational Thinking (ENACT) Approach

6. 1850-0952 v.6 Regional Educational Laboratory, Midwest Usability and Feasibility Study for the Data-Informed Leadership for Equity (DILE) Approach

7. 1850-0952 v.7 Regional Educational Laboratory, Midwest Usability and Feasibility Study for the Making Equitable Schools Audit (MESA) Approach

3. Use of Information Technology

To the extent possible NCEE will use automated data collection techniques, such as when the survey or assessment being tested employs automated methods for its data collection or to assess the utility of machine learning for qualitative data coding. This clearance offers NCEE the opportunity to try innovative technologies that would reduce burden and increase the use of information technology.

4. Efforts to Identify Duplication

Information collection and analysis under this clearance will not duplicate any other feasibility and design work being conducted by NCEE or other federal and non-federal agencies that may collaborate in this work. Additionally, to the extent possible, NCEE will make use of existing information, including reviewing results of previous evaluations of instruments, methods, and interventions to build upon past efforts.

5. Minimizing Burden

This research will be designed as relatively small-scale data collection and design efforts so as to minimize the amount of burden required to improve questionnaires and procedures, test new ideas, and refine or improve upon positive or unclear results from other tests. The results of the research conducted under this clearance are expected to inform and improve the interventions, methods and instruments utilized in full- scale studies and thereby improve information quality while minimizing burden to respondents.

6. Consequences of Less Frequent Collection

Without the exploratory and design tests proposed as part of this collection, the quality of the full experiments, quasi-experiments, and implementation evaluations would suffer, as would our ability to design studies in a timely manner to meet policy goals.

7. Special Circumstances

There are no special circumstances.

8. Consultations Outside the Agency

A 60-day Federal Register notice for this request published on April 25, 2023 (88 FR 24980). There were no public comments received. IES will publish a 30-day Federal Register notice to solicit additional public comments.

Consultation with staff from other Federal agencies that sponsor surveys conducted by NCEE will occur in conjunction with testing individual surveys. Consultation with staff from other Federal cognitive laboratory facilities may also occur as part of joint exploratory research efforts. These consultations will include discussions concerning potential response problems, clarity of questions and instructions, and other aspects of respondent burden. Additional efforts to consult with potential respondents to obtain their views on the availability of data, clarity of instructions, etc., may be undertaken as part of the testing that is conducted under this clearance.

9. Paying Respondents

Respondents for activities conducted in the laboratory (e.g. cognitive interviews and focus groups) under this clearance may receive compensation for travel and participation. This practice has proven necessary and effective in recruiting subjects to participate in such research and is also employed by the other federal cognitive laboratories. Research on incentives that may be conducted under this clearance may also involve nonmonetary incentives. The Office of Management and Budget (OMB) has noted that effectiveness of such incentives is a worthwhile research topic. If incentives need to be proposed for any research activity under this clearance, justification will be provided and NCEE will work closely with OMB on the incentive strategy to be employed. NCEE will typically propose incentives at the level approved by OMB for cognitive laboratories and focus groups. If a higher incentive amount is proposed for approval, a meaningful justification will be provided.

10. Assurance of Confidentiality

If the collection is under the authority of Education Sciences Reform Act of 2002 (ESRA 2002), all respondents who participate in research under this clearance will be informed that their participation is voluntary and that all of the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). For personal visit and telephone interviews, this information will be conveyed verbally by the interviewer, and in personal visit interviews respondents will also receive this information in writing. For self-administered questionnaires, the information will be included in the mailing package, either as part of communication materials or on the questionnaire or instructions. For Internet-based data collections, this information will be displayed prominently, and in a format that allows the respondent to print it out. All participants in cognitive research will be required to sign written notification concerning the voluntary and confidential nature of their participation. NCEE will also inform respondents in writing of the need to have an OMB number. No participant direct identifiers will be maintained as part of research under this generic clearance.

11. Justification for Sensitive Questions

Questions that are included in NCEE data collections are typically not of a sensitive nature and should not pose a problem to respondents. However, it is possible that some potentially sensitive questions may be included in instruments that are tested under this clearance. One of the purposes of the testing is to identify such questions, determine sources of sensitivity, and alleviate them insofar as possible before the actual survey is administered.

12. Estimate of Hour Burden

We estimate that the number of people involved in our exploratory, field test, pilot, cognitive, and focus group work will be at most 6,000 per year, the vast majority of which will be contacted as part of screening and recruitment activities preceding the actual research. Given that screening and recruitment activities are included in the burden calculations, we estimate the annual burden hours will be approximately 0.5 hours per person or an annualized 3,000 hours overall. The total estimated respondent burden is 9,000 hours for the 3-year period beginning on the date of OMB approval in 2023:

Time Period	Respondents	Responses	Respondent burden (hours)
2023 – 2024	6,000	6,000	3,000
2024 – 2025	6,000	6,000	3,000
2025 – 2026	6,000	6,000	3,000
Total	18,000	18,000	9,000

A variety of forms will be used in conducting the research under this clearance, and the exact number of different forms, length of each form, and number of subjects/respondents per form are unknown at this time. Similarly, the exact respondents are unknown but are likely to include students, teachers, parents, schools, local and state agency officials.

We estimate that the average number of generic information collections that OMB may expect to receive under this clearance is approximately 15 submissions each year.

13. Estimate of Cost Burden

There is typically no cost to respondents for participating in the research being conducted under this clearance, except for their time to complete the requested activity.

14. Cost to Federal Government

There is no way to anticipate the actual number of participants, duration, and/or mode of data collection to be conducted under this clearance. Thus, it is impossible to estimate in advance the cost to the Federal Government. Costs will be covered by divisions conducting the research from their data collection budgets. We will include information about costs in the individual submissions.

15. Reason for Change in Burden

NCEE anticipates more preliminary and exploratory activities in the next three years that will inform and guide important evaluation and research efforts. During the time of the initial clearance in 2020, NCEE was primarily completing projects that had already been approved by OMB or were affected by the global pandemic. NCEE is in a new phase of developing research agendas in all of its main content areas after clearing out the previous studies that began prior to or during the pandemic period. This will require more exploratory work in order to develop higher quality studies. We anticipate developing a set of new studies that will require these key preliminary activities.

The expected change in the annual burden hours is 1,000, and the expected change in the annual number of responses is 2,000, based on a program change due to agency discretion:

	Program Change Due to New Statute	Program Change Due to Agency Discretion	Change Due to Adjustment in Agency Estimate
Annual Burden Hours		1,000
Annual Responses		2,000
Annual Costs (if applicable)		Not applicable

16. Project Schedule

This research program is for study design, instrument and procedure development purposes. Data tabulations will be used to evaluate the results of intervention, questionnaire, and methods testing. The information collected in this effort will typically not be the subject of population estimates or other statistics in NCEE reports; more often, it may be published in research and development reports or be included as a methodological appendix or footnote in a report containing data from a larger data collection effort. The results of this research may be prepared for presentation at professional meetings or publication on NCEE website and in professional journals. Due to the nature of this clearance, there is no definite or tentative time schedule for individual testing activities at this point. We expect work to continue more or less continuously throughout the duration of the clearance.

17. Request to Not Display Expiration Date

No exemption is requested.

18. Exceptions to the Certification

There are no exceptions to the certification.

1 While NCEE is the sponsor of this generic clearance, there may be occasion for other Centers within IES, such as the National Center for Education Research (NCER) and the National Center for Special Education Research (NCSER), to use the system clearance for similar purposes.