WWC Customer Satisfaction Survey RELs

WWC Customer Service Focus Groups:

Protocol for REL Meeting Focus Groups

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB number. The valid OMB control number of this information collection is 1800-0011. The time required to complete this information collection is estimated to average 75 minutes per customer satisfaction focus group. If you have any comments concerning the accuracy of the time estimates(s) or suggestion for improving this form, please write to: U.S. Department of Education, Washington, 20202-4651. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: Institute of Education Sciences, US Department of Education, 555 New Jersey Ave, NW, Washington, DC 20208.

A. Consent/introduction to the project

Thanks again for taking the time to speak with me this morning/afternoon. Before we start, I’d like to provide a little background on our work, and answer any questions you might have for me. As you probably know, I work for Mathematica Policy Research, which is a nonpartisan research organization. Mathematica is one of the contractors operating the What Works Clearinghouse (WWC), which is funded by the Institute of Education Sciences (IES). The mission of the WWC is to be a central source for evidence from high quality research in order to answer ‘what works’ in Education to improve student outcomes. We are conducting a formative evaluation on the What Works Clearinghouse (WWC). As part of the evaluation, we will be conducting customer satisfaction focus groups like this one with 5 Regional Education Laboratories (RELs) throughout the country. This short conversation will ask you first to comment on your familiarity with the WWC in general and then ask you to engage and react to 3 specific products that the WWC recently produced. We anticipate using the ideas this group generates to improve WWC products and services.

We will NOT report results in any way that would identify you. Your participation is voluntary, and you do not have to respond to any questions you do not want to. 

I’m recording the discussion so I don’t have to take detailed notes and can listen carefully to what you are saying. The recording is just to help me remember what you say. No one outside of the research team will have access to the tape. Do we have your permission to record? Do you have any questions before we begin?

Let’s get started. [HIT THE RECORD BUTTON].

I have hit the record button.

_______________________________________________________________________

B. Introductions

• Thank you for taking just a few moments to tell me about yourself. This will help inform our conversation and provide me with some context as to your background. Just so I can put a name to a face, can you please go around the room and tell us your first and last name?

Today I’ll be talking with you about a few topics. These include: your familiarity with the What Works Clearinghouse, and what products or types of information may be helpful to the practitioners you work with. Then we will spend some time reviewing specific What Works Clearinghouse products in order to gather your thoughts on the utility of the products and how they could be improved.

Links to these products were sent in advance of the meeting for your review but if you haven’t reviewed them, that is perfectly fine.

___________________________________________________________________________

C. Familiarity with the WWC

1. [For those of you who have heard of the What Works Clearinghouse] Have you ever visited the website?

1. If so, what was your motivation for visiting the What Works Clearinghouse?

2. Did you use the information presented on the website to inform decision making? If so, what decisions?

3. Were you able to find what you were looking for?

• If so, was it easy to find the information? Was the site easy or hard to navigate?

• If not, what would help you find that information on the Website?

2. [For those of you who have heard of the What Works Clearinghouse but never visited the What Works Clearinghouse] Why haven’t you visited the website? (i.e., no time, didn’t think you’d find what you needed there?)

3. [For all]What is your understanding of the resources available through the What Works Clearinghouse? Who do you feel the target audience(s) is/are?

[Note: During this part of the discussion, the interviewer will have examples of the intervention report ratings on hand for reference.]

4. Thinking about classroom teachers as a target audience:

1. What types of questions do teachers seek to answer when they visit the WWC?

2. Are the report ratings easy to understand to teachers?

(If difficult to understand) What changes can you suggest to report ratings to make them more understandable to teachers?

5. Reflecting now on administrators as a target audience:

1. What types of questions do administrators seek to answer when they visit the WWC?

2. Are the report ratings easy to understand to administrators?

(If difficult to understand) What changes can you suggest to report ratings to make them more understandable to administrators?

6. Reflecting now on researchers as a target audience:

1. What information and or materials are most useful?

2. What additional materials could be useful?

_____________________________________________________________________________

I’d like to now take about 20 minutes to review three of the products that the What Works Clearinghouse has recently produced. We are very interested in getting your feedback as to their clarity and utility to better inform future products. After we engage these products, we’ll have a series of questions for you.

______________________________________________________________________________

D. Product Number 1: Video

The What Works Clearinghouse has produced one video on the ways that WWC resources can help with the Selection of Math Instructional Materials. This video is intended to provide district-level decision-makers with information on what WWC resources can inform the selection process. This is a short, 4 minute video that I’m going to show you now.

[Show Sample Video (4 minutes): “How to Select Mathematics Instructional Materials”]

7. Let’s talk a bit about the video.

1. Is this video useful for district decision makers?

2. Is the content in the video clear? If not, in what ways could it be clearer or improved?

3. Would you recommend this particular video to others? If so, whom? If not, why not?

4. What changes, if any, would you make about the information the video contains and/or how the information is presented? Do you have suggestions for how the product could be improved?

5. Is the length of the video appropriate?

6. Are there other topics for videos that you could suggest that might be useful to you and/or other practitioners you work with?

_____________________________________________________________________________

E. Product number 2: A sample practice guide summary/By the Numbers

The WWC produces “practice guides” which provide evidence-based recommendations for educators on a range of topics. The recommendations are made by a panel of experts after a review of the research. Each recommendation includes implementation steps and solutions for common roadblocks. To complement these often lengthy guides, the WWC has started to produce “practice guide summaries” and/or “by the numbers” products which synthesize the key recommendations from practice guides.

[Interviewer to demonstrate/walk through the product on a laptop with a centralized projector]

8. Let’s spend a few minutes now talking about the practice guide summary/by the number document.

1. What information in the summary did you find informative and useful?

2. Is the content clear? If not, in what ways could it be improved?

3. Would you recommend it to others? If so, whom? (i.e. would you show it to your school leaders? District staff?) If not, why not?

4. Does it pique your interest in the full guide?

What additional materials, if any, would make the summaries/By the Numbers product more useful?

___________________________________________________________________________

F. Product number 3: Topical Blast

The WWC also produces web pages and corresponding email blasts that consolidate information and resources from the What Works Clearinghouse related to a specific topic or event. Those who do not receive the email blast can access the materials in the ‘rotator’ at the top of the website or in the NEWS & EVENTS feature. The WWC has produced a handful of these blasts over the past year. They include information on going to college, challenging classroom behaviors and literacy, among other topics. I will now guide you through the Back to School blast, which was released in late August to coincide with the start of the academic semester. This consolidates many WWC products on instruction.

[Interviewer to demonstrate/walk through the product on a laptop with a centralized projector]

9. Let’s spend a few minutes discussing the topical blast we just explored.

1. What information did you find informative and useful?

2. Is the content presented in the topical blast clear?

3. Would you recommend it to others? If so, whom? If not, why not?

4. What changes, if any, would you make about the information the blast contains and/or how the information is presented? Do you have suggestions for how the product could be improved?

5. Are there other topical blasts that you could suggest that might be useful to you and/or other practitioners you work with?

__________________________________________________________________________

G. Dissemination

10. You have been able to interact with several of the products this [afternoon/morning].The What Works Clearinghouse currently engages in quite a few dissemination strategies. In addition to the email blasts we mentioned, the WWC posts materials on the website for users to find via other channels and uses facebook and twitter to alert users to new content.

a) Are there audiences that you feel the WWC (IES) may not be targeting? Which ones? What type of information would they want to access?

b) Do you have any overall suggestions for ways in which the information on the WWC could be disseminated? Do you have any suggestions of how to get new users to the WWC?

11. While we have viewed three products, (a video, a practice guide summary and a topical blast) are there other formats (i.e. types of reports, etc) you think the What Works Clearinghouse should produce?

a) What formats would be useful? Why?

_____________________________________________________________________________

H. Wrap Up

12. As we close, I just wanted to remind you that the formative evaluation is working to evaluate ways in which the WWC and its associated products could be improved. With this in mind, is there anything that I haven’t asked you about regarding the What Works Clearinghouse and your work with the REL that you’d like to comment on? Is there anything else we should be aware of?

DRAFT 4