Justification Memo

DOCUMENTATION FOR THE GENERIC CLEARANCE

OF CUSTOMER SERVICE SATISFACTION COLLECTIONS

TITLE OF INFORMATION COLLECTION:

Customer Satisfaction Survey of the Relevance and Utility of the What Work Clearinghouse (10-day review request)

[X ] SURVEY [ ] FOCUS GROUP [ ] SOFTWARE USABILITY TESTING

DESCRIPTION OF THIS SPECIFIC COLLECTION

1. Intended Purpose

This submission is a request for approval of data collection activities to conduct a customer satisfaction survey of users of the What Works Clearinghouse (WWC) website (www.whatworks.ed.gov) and its products. The mission of the Clearinghouse is to provide educators, and policymakers with a central and trusted source of scientific evidence for what works in education.

The purpose of conducting the customer satisfaction survey is to assess the ongoing information needs of WWC website users and potential users, and to identify areas for improvement to increase the relevance and usefulness of WWC products.

2. Need for the Collection

Recognition continues to increase among policymakers that decisions about educational programs, products, and practices should be based on evidence of effectiveness from high quality scientific research. The No Child Left Behind Act (PL 107-110), for example, includes numerous requirements to use the findings from “scientifically based research” in designing and implementing educational interventions. The implementation of this Act – with its promise to serve the educational needs of all children, and its emphasis on “doing what works” – creates a very strong impetus to identify, select, and implement effective school improvement strategies.

In order to efficiently and effectively meet the challenge and achieve the promise of No Child Left Behind depends on educators, education administrators, and parents having timely access to reliable information to support informed decision-making. Those who have a stake in improving education want credible and reliable information that allows them to judge the effectiveness and quality of alternative approaches to improving outcomes for students. By transforming education into an evidence-based field, the U.S. Department of Education aims to fultill its mission of improving student outcomes and providing the information that decision makers need.

The What Works Clearinghouse is a key part of this effort. Established by the U.S. Department of Education’s Institute of Education Sciences in 2002, the purpose of the WWC is to offer everyone – from elected officials to the public at large – a central, independent, and trusted source of scientific evidence of what works and does not work in education. To be successful, the WWC must assess the ongoing needs of its customers and potential customers regarding the Clearinghouse’s products and services, and involve key constituents in the process of its activities, in the development of its products, and in identifying areas for improvement.

The WWC requests OMB clearance to administer a Survey of Customer and Potential Customer Satisfaction and Needs. The online surveys are a means for the WWC to monitor its services to ensure that it produces and provides relevant online products and services that allow end users to more effectively improve educational outcomes for children. The voluntary survey will gather information from WWC website users and potential users about the level of their satisfaction with the online services and information provided by the WWC. WWC and ED staff will use the resulting survey data to enhance the structure and content of the website and to ensure that it is tailored to user needs and is easy for users to navigate.

3. Planned Use of the Data

The data gathered by the survey will be used by the WWC and ED staff to monitor WWC products and services to ensure that it produces and provides relevant online products and services that allow end users to more effectively improve educational outcomes for students.

4. Date(s) and Location(s)

Data will be collected during the second and third quarters of 2011. The following table shows the time line for this study.

sCHEDULE OF aCTIVITIES

Activity	Schedule
Prepare list of potential report users	March – May 2011
Conduct survey	April – May 2011
Analysis and report of findings	June 2011

5. Collection Procedures

The WWC contractor, Mathematica Policy Research, Inc., will contact current and potential users of WWC publications. Potential respondents will receive an email invitation to participate in the survey (Attachment B). The invitation will include a link to the short, web-based questionnaire, which will take respondents no more than 10 minutes to complete. To improve the response rate, non-respondents will be sent a follow-up email request two weeks after the initial email invitation.

The data collection plan reflects sensitivity to issues of efficiency, accuracy, and respondent burden. To conduct the WWC survey, we will use a web-based data collection method programmed to accept only valid responses and to check for logical consistency across answers. Respondents will be able to correct any errors or clarify inconsistent data. An added advantage of web-based data collection is that respondents can complete the survey at their convenience. All communications with potential respondents will include a project-specific email address and a toll-free telephone number to use if potential respondents have questions.

This is a voluntary survey, as stated in the invitation, and respondents may choose whether or not to participate. Also, they may choose to not answer specific questions included on the questionnaire. All of the responses from the survey will be kept strictly confidential to the extent allowed by law. The project will adhere to industry standards to protect survey participants to the extent the law allows. No names will be attached to any responses. Mathematica has rigorous requirements in place to protect the security of the electronic responses. The following statement will be used to describe data confidentiality to respondents:

Responses to this questionnaire will be used only for statistical purposes. The reports prepared for this study will summarize findings across the respondents and will not associate responses with a specific individual. We will not provide information that identifies you to anyone outside the study team, except as required by law.

6. Number of Focus Groups, Surveys, Usability Testing Sessions

This request is for one data collection – a web survey. Up to nine people will be contacted to pre-test the web questionnaire.

7. Description of Respondents/Participants

The target population of this survey includes two groups. The groups are:

1. Those who currently receive WWC e-mail updates

2. Those who attend conferences where the WWC has an exhibit table and visit the table

The contractor currently maintains a list of 6,068 individual email addresses. These email addresses receive periodic updates regarding WWC publications and are considered current users who are familiar with the WWC.

Contractor staff will attend five education-related conferences from 2011 on behalf of the WWC. Attendees at these conferences will be educators, education decisions makers, and policymakers. Individuals who visit the WWC Exhibit Hall Booth during these conferences will be provided with an overview of the WWC and asked to provide their email address to complete a short survey about the WWC. Based on previous years’ experience, Mathematica can expect to receive approximately 425 email addresses (85 from each conference). These individuals are considered potential users who may not be familiar with the WWC and its publications.

All email addresses receive an invitation to participate in the short, web-based questionnaire.

STATISTICAL INFORMATION

The list of those who currently receive WWC e-mail updates and those who attend conferences where the WWC had an exhibit table and visited the table will be used in their entirety (approximately 6,493 e-mail addresses total). The list will be checked for duplicate e-mails and if any are detected, one record will be randomly selected to be retained.

The contractor expects a 35 percent response rate for two reasons.

1. People who “screen out” because they are not aware of any What Works Clearinghouse publications will be considered completed cases. There is minimal burden for these respondents since they will be asked few questions.

2. For online surveys in which there is no prior relationship with respondents, response rates can be as high as 30 percent. The What Works Clearinghouse already has a prior relationship with 6,068 potential respondents, as they receive e-mail updates.

The contractor therefore estimates that approximately 35 percent of respondents will complete the survey.

AMOUNT OF ANY PROPOSED STIPEND OR INCENTIVE

No financial incentives or gifts will be offered to respondents.

BURDEN HOUR COMPUTATION – POTENTIAL RESPONDENTS

(Number of respondents (X) estimated response or participation time = burden hours)

Category of Respondent	No. of Potential Respondents	Participation Time*	Burden
Current User; Full Questionnaire	6,068	0.17 hour	1,031.56 hours
Non-User; Partial Questionnaire	425	0.12 hour	51.0 hours
Total	6,493 potential respondents		1,082.56 hours

*Average time estimated 6,068 x .17 hour (anticipated number of users; full questionnaire, 11 questions, 10 minutes) and 425 x .10 hour (anticipated number of non-users; partial questionnaire, 8 questions, 7 minutes).

BURDEN HOUR COMPUTATION – ANTICIPATED RESPONDENTS

(Number of respondents (X) estimated response or participation time = burden hours)

Category of Respondent	No. of Anticipated Respondents*	Participation Time**	Burden
Current User; Full Questionnaire	2,124	0.17 hour	361.08 hours
Non-User; Partial Questionnaire	149	0.12 hour	17.88 hours
Total	2,343 anticipated respondents		378.96 hours

*As indicated in the “Statistical Information” section, the Contractor expects a 35% response rate.

**Average time estimated 6,068 x .17 hour (anticipated number of users; full questionnaire, 11 questions, 10 minutes) and 425 x .10 hour (anticipated number of non-users; partial questionnaire, 8 questions, 7 minutes).

BURDEN COST COMPUTATION

There are no cost burdens to respondents.

REQUESTED APPROVAL DATE: May 2, 2011

NAME OF CONTACT PERSON: Susan Sanchez

TELEPHONE NUMBER: (202)208-7061

MAILING LOCATION: IES, CP

ED DEPARTMENT, OFFICE: IES

6