ED-OMB Q&A December 2009

MEMORANDUM

DATE  :     December 15, 2009

TO    :     Bridget Dolling, OMB

CC    :     Kate Mullan, Kathy Axt; James Hyler, Janice Ballou   

FROM  :     Amy Feldman Farb/Barbara Vespucci   

Subject:    Responses to OMB Question on 1800-0011: Evaluation of the Relevance and Utility of National Center for              Education Evaluation (NCEE)      

Below are ED/IES’s responses to the questions from the Office of Management and Budget regarding the Survey of the Regional Educational Laboratory (REL) Research Reports.

1.    Supporting Statement

QUESTION

1a.   Please provide some background on why a 70% response rate is expected, or revise that rate with explanation.

ANSWER

The questionnaire is designed to answer two key questions: (1) Do the members of the sample populations know about the Fast Response series of reports? and (2) For the reports they do know about, what is their assessment of the reports’ relevance and usefulness?  We expect a 70 percent response rate for two reasons.

The first reason is that people who “screen out” because they are not aware of any Fast Response reports will be considered completed cases. There is minimal burden for these sample members since they will be asked few questions. The second reason is we have been averaging about a 60 percent response rate on the Bridge Event Survey. The sample for that survey is attendees at REL events, which is one of the populations to be sampled for the Survey of the REL Research Reports. We have obtained this response rate with a data collection design in which the amount of time available for conducting follow-up calls with nonrespondents is limited, in order to minimize recall error. For the Survey of the REL Research Reports, we will not have these time constraints and will therefore have more time to recontact nonrespondents and achieve a higher response rate.

QUESTION

1b.  Please explain what non-response bias analysis/adjustments are planned.

ANSWER

To develop the sample frame, we identified four groups as the target population. The information available to conduct the non-response bias analysis differs for these four groups and most, if not all, of that information will be used in stratification. Therefore, for the nonresponse bias analysis we will examine response rates by sampling strata, and if REL is not used in sampling, by REL. For the CSSOs, we will examine nonresponse by REL. For each of the four groups, we will make weighting adjustments to correct for differences. This information will be presented in a table with a memo describing the nonresponse characteristics.

QUESTION

1c.   Please state whether this is a voluntary or mandatory survey, and please explain how the information will or won't be treated with confidentiality.  Please add a description of these two issues to the supporting statement, email invitation, and FAQ.

ANSWER

This is a voluntary survey and sample members may choose whether or not to participate. Also, they may choose to not answer specific questions included on the questionnaire. All of the responses from the survey will be kept strictly confidential.  The project will adhere to the guidelines outlined in the Mathematica Security Manual to protect survey participants to the extent the law allows. Sample members will be assigned unique identification numbers to ensure confidentiality, and no names will be attached to any responses. Mathematica has rigorous requirements in place to protect the security of the electronic and paper survey responses.

Revisions will be made to the email invitation and the FAQ to include this information.

QUESTION

1d.  Related, the confidentiality statement needs to be fixed – they should use the typical NCEE and/or RELs language if this is being collected under ESRA.

ANSWER

The following language will be added to the FAQ document and the email request to participate in the survey:

Responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific individual. We will not provide information that identifies you to anyone outside the study team, except as required by law.

2.    Instrument

QUESTION

2a.   B2a Should there be a category “I have not heard of this report?” It’s not clear if the responder will acknowledge hearing of the report prior to answering B2a.

ANSWER

B2a will include only reports the respondents have heard of. While it may not be clear on the paper version of the questionnaire, the web programming will be set up so only reports that respondents identify that they “are familiar with” in B1 will be pre-filled to be answered in B2a. 

QUESTION

2b.  B5 “scientifically gathered and presented” seems like a strange label.  Would something like “reflective of high quality science,” “credible and based on science,” or “scientifically valid” or some combination of these be more appropriate?

ANSWER

We had thoughtful discussions about the best wording to use to measure the scientific attributes of the reports. Our approach is to cognitively test the current wording during the upcoming pretest. We will also review these alternate wording suggestions with pretest respondents. We will then make recommendations for final question wording.

QUESTION

2c.   Expand C1:

2ci. - Government to include “other”

2cii. - Education to include “Early Childhood Educator,” “paraprofessional,” “School Psychologist,” “Special Education Teacher or Related Service provider”

2ciii. - Other to include “parent”—if we think parents will be respondents and can be included through sampling options 2 and 3.  While not the primary audience for the reports, it would be interesting to see to what extent parents are consumers of the products and whether they find the products useful.

ANSWER

2ci. Suggestion for this change will be made.

2cii. We suggest that, instead of adding other categories that will extend the length of the list, we include “please specify” after the following response choices: (#5) School district central office staff, (#7) Other school-level administrator, (#9) Teacher, and (#11) Curriculum specialist, to collect additional information about the particular focus of respondents in these areas. In the analysis, we will consider whether any specific classifications occur in sufficient numbers to add them as separate categories.

2ciii. Including parents may be confusing since our respondents may also be parents in addition to their professional role. There is a general “Other” category in which respondents who are only parents and do not fit into the other categories can provide that information.

QUESTION

2d.  C2:  Include “early childhood education setting (e.g. Head Start, preschool),” “home school (or add home school to the ()’s after “including vocational high schools”)

ANSWER

We will add the response choice “early childhood education setting” to the C2 list of businesses and organizations and include “home school” in the description of (#7) Elementary or secondary school.

3.    FAQ

QUESTION

3a.   How will respondents access this document?

ANSWER

Respondents will be able to access the FAQ document on every page of the web questionnaire; clicking on a button at the bottom of the page will display the FAQ. The email invitation to participate in the survey will inform respondents that the FAQ document is available on the questionnaire website, which they can access using the link we will provide. We will also provide an 800 telephone number and a survey email address that respondents can contact to request the FAQ document if they are unable to access it on the web questionnaire.

QUESTION

3b.  Please remove the extra period after the response to “What are the benefits to participating?”

ANSWER

The document has been edited to remove the extra period.