OMB Responses

U.S. Department of Education

Office of Vocational and Adult Education

Strengthening America’s Competitiveness through Adult Numeracy Instruction

Responses to OMB Questions

December 1, 2010; Revised February 2, 2010

MPR’s answers to the questions from OMB follow each re-statement of the question below.

Questions/Clarification

1. Please provide a summary of the full scale information collection plans to help us understand which components the field test will inform and which it will not, what its coverage and sample size will be, and what its goals will be.

The goals of the field test are to assess the implementation of the Adult Numeracy Instruction professional development model and its effects on teaching in adult education programs. The field test will provide information about current teaching practices and the mathematics content knowledge of a sample of teachers from two states and changes to teaching practices that occur as a result of participation in the professional development. In addition, it will provide information about the effectiveness of various aspects of the professional development and will serve to guide revisions to the model. The table below presents the various components of the model, their projected implementation dates, and the data collection methods. With the exception of the classroom observation that is part of the professional development, the instruments used for data collection have been submitted as part of the OMB package. To participate in the professional development field test, states will submit applications, and two states will be chosen. From each of those states, 10 administrators and 20 teachers will be identified to participate.

Professional Development Activities and Field Test
Component	Timeframe	Data Collection
State recruitment	January – August 2010	State applications
Classroom observations	September 2010	Observation guidelines
Institute 1	October 2010	Teacher pre-questionnaire Administrator pre-questionnaire Teacher pre-cognitive assessment
Implementation of lessons and local/regional participant meetings	October – December 2010	-
Institute 2	December 2010	-
Implementation of lessons and local/regional participant meetings	December 2010– March 2011	-
Institute 3	March 2011	Teacher post-questionnaire Administrator post-questionnaire Teacher post-cognitive assessment
Classroom observations	March/April 2011	Observation guidelines Teacher interviews

2. Related to the 11 “research questions” on pages 6-7 of the SS:

a.    What is the state of knowledge for each of these (i.e., are these questions for which there is little or no literature)?

b.    Please map these to items in the questionnaires and assessments so we can see how you intend to inform them via the field test and full scale study.

We included these questions in Supporting Statement A because they reflect a comprehensive project of which the field test of the professional development is a part. We believed it would provide context for the field test. However, the questions that are listed are to be answered by other tasks, particularly a review of the literature. In order to be more clear, we have now included only questions to be answered by the field test. These are mapped to the cognitive assessments and to items on the questionnaires.

Questions to be answered by Field Test mapped to items on Cognitive Assessments and Teacher and Administrator Surveys:

Question:	Items on instruments
What do teachers learn from the institutes about concepts related to Algebra, Geometry, Numbers and Operations, and Data, Probability, and Statistics?	CA1 and CA2
What effect does the professional development have on teachers’ confidence in teaching math?	TQ1: Items 7, 8, 9, 10; TQ2: Items 1, 2, 3, 4
What effect does the professional development have on teachers’ plans for using specific instructional practices and on their practices in the classroom?	TQ1: Items 24-25; TQ2: Items 5-6; Observations
What effect does the professional development have on teachers’ perspectives on learning and teaching math?	TQ1: Items 26-27; TQ2: Items 7-8
How does the professional development affect teachers’ need for support, i.e., the kind of support they need?	TQ1: Item 28; TQ2: Item 9
What are teachers’ expectations for the institutes and how well do they feel the goals were met?	TQ1: Items 29-31; TQ2: Items 10-12
What specific benefits do teachers report for the professional development, and what improvements do they suggest?	TQ2: Items 13-17
What effect does the professional development have on administrators’ knowledge of math and comfort in supporting teachers?	AQ1: Items 6-7; AQ2: Items 1-2
What effect does the professional development have on administrators’ awareness of instructional practices and perspectives on teaching math?	AQ!: Items 18-19, 21-22; AQ2: Items 3-6
What effect does the professional development have on their plans for supporting teachers in their programs?	AQ1: Item 20; AQ2: Item 7
What are administrators’ expectations for the institutes and how well do they feel the goals were met?	AQ1: Items 23-25; AQ2: Items 8-10
What specific benefits do administrators report for the professional development, and what improvements do they suggest?	AQ2: Items 11-15

*Key: CA1= Cognitive Assessment pre-institute; CA2=Cognitive Assessment post-institute; TQ1= Teacher Questionnaire; TQ2=Teacher Questionnaire post-institute; AQ1=Administrator questionnaire

3. How likely is it that the teachers and administrators in the program will receive professional development (e.g., mentoring or other courses) in the interval between the pre-test and the post-test?  How will the evaluation take those external events into account in measuring the “impact” of the program?

Resources for providing in depth professional development for adult educators are very limited, so it is very unlikely that participants would receive any additional professional development in the six months’ time between pre-test and post-test. We could, however, add a question regarding their participation in any other professional development to the teacher post-questionnaire if it is felt that it is important to do so.

4. Did the Department consider identifying a control group?

Yes, this was considered. It is, however, very difficult to identify a group for this purpose that could be considered comparable. While it may be important to ascertain whether a comparison group would demonstrate increased knowledge of math content or of specific pedagogical practices taught in this professional development program just from their day-to-day professional experiences, it was deemed to be too costly and burdensome to have them complete pre- and post- assessments and questionnaires that are unlikely to show much, if any, difference because there is a short time span between the pre- and post-instruments and because the learning is very specific to the professional development.

5. Did the Department consider measuring directly some of the primary concepts in the post-test, such as administrators’ self-assessment of program benefits and teacher intention to change instructional practices

Consideration was given to measuring directly the primary concepts in the post-test, however, given the purpose of the field test (to revise a professional development model), the size of the field test sample, the lack of available instrumentation to measure these constructs directly, and the limited project resources, it seemed more strategic to maximize resources to collect more descriptive feedback from participants. This information will provide what is needed to make post-field test revisions to the professional development intervention.

6. Please clarify the field test schedule in relation to the professional development institute sessions schedule mention in A6 of the SS (i.e, that have already occurred).

The time it took to assemble, submit, and conduct reviews of the supporting documents resulted in a need to change the timeline that was not anticipated. As a result, A6 of the revised SS now reads as follows:

Data will be collected at various intervals during this study. The frequency of data collection varies by respondent. Teachers will receive the survey and cognitive assessment at the first and after the third professional development sessions. Program administrators will receive the survey at the first and after the third professional development sessions. Teachers will be interviewed at the end of the third professional development session and at the end of the school year following the first institute. The institutes will take place in October and December of 2010 and March of 2011. The teachers will participate in the last follow-up interview during March/April 2011. (See table for Question 1)

7. Please provide the formal reference for the “Obey-Porter law (1987)” mentioned in A10.  Also, please clarify if that is the statutory authority for this collection (if so, please provide a specific citation).  Finally, is confidentiality assured under that statute?  If so, please provide the specific citation for the confidentiality provision. 

When the SS was prepared, MPR used language from an earlier OMB submission that is now outdated. The following is a current statement describing security procedures:

MPR Associates has a robust program in place to preserve the confidentiality of information received from our clients, who reflect a wide range of public- and private sector organizations. These procedures are designed to comply with all requirements of the Privacy Act of 1974, 5 U.S.C. § 552a, Public Law No. 93-579, which imposes strict constraints on the use of individually identifiable data and serious consequences for the unlawful disclosure of these data. Since its founding 35 years ago, MPR has never had an instance of unauthorized disclosure of confidential information. In part, this record is due to procedures guaranteeing that only authorized employees have access to computing equipment and data files. Equally important, however, is a corporate culture supportive of client confidentiality through formal company policies and by fostering an atmosphere conducive to the highest degree of professional responsibility. All staff signs a statement describing their responsibility for maintaining confidentiality. MPR’s experience with maintaining confidentiality and restricted access to confidential materials extends to various government agencies and private clients as well.

The instructions for the questionnaires to be administered as part of the field test indicate that responses will only be reported in summary form. MPR Associates Inc. is a licensed user of individually identifiable data from the National Center for Education Statistics (particularly Computer Security Act of 1987). MPR adheres strictly to the requirements IES Statistical Standards Program, which imposes strict constraints on the use of individually identifiable data and serious consequences for the unlawful disclosure of these data. In addition, it is standard practice at MPR to separate survey responses from respondent identifying information, to assign a random respondent ID, and to maintain the cross-walk between the random ID and the identifying information in a secure, locked location. These requirements go above and beyond federal requirements. MPR has always received the highest marks for data confidentiality procedures during site visits for compliance by the federal government.

7. Teacher questionnaire – how would a teacher know about the percentage of adults in a class diagnosed with a learning disability?  Do teachers have access to medical records?  Are students asked to provide such information at enrollment?

Teachers often do know about some of their students that have disabilities, either because it is in their records or because students have requested special services. We thought it was important to ask this question because teachers so often mention it as a challenge in their teaching. However, because we can’t know how accurate the information they provide would be, we have decided to delete this question.

4