NRS State Data Quality Checklist

NRS State Data Quality Checklist

State: Date:

C ompleted by (name and title):

A. Data Foundation and Structure

Acceptable Quality

1. State has written assessment policies that specify: Yes No

• Standardized assessments to use for accountability that are valid and appropriate for adult students.

• Time periods (in hours or weeks) for when to pre- and posttest.

• Score ranges tied to educational functioning levels (EFL) for placement and for reporting gains for accountability.

• Appropriate guidance on tests and placement for special populations (e.g., students who are unable to be tested due to language or disability).

• Unacceptable methods of assessment for EFL placement.

• Appropriate guidance on requirements and conditions for testing distance education students reported in the NRS.

1a. List up to three of the most of commonly used assessments permitted for ABE and ESL.

ABE Assessments:

ESL Assessments:

2. State has written policies for follow-up that explain: Yes No

• Goal setting procedures.

• Follow-up methodology (survey or data match) for each measure that meets NRS requirements.

• Which of students are to be followed.

• Difference between goal setting for NRS and goals for instruction.

2a. Indicate your follow-up methods for each measure.

Entered employment: Survey Data match Both (explain)

Retained employment: Survey Data match Both (explain)

Obtain GED: Survey Data match Both (explain)

Enter postsecondary: Survey Data match Both (explain)

3. If state uses survey follow-up method for any measure (check not applicable (NA) and skip to the next item if survey is not used): NA

• Local programs or state can produce a list of students to survey, according to NRS requirements. Yes No

• Survey is conducted with a state provided, standard survey instrument. Yes No

• State has a regular schedule (e.g., quarterly) for submission of survey data or student names from local programs. Yes No

4. If state uses data matching for any measure (check not applicable (NA) and skip to the next question if data matching is not used): NA

• Local or state data system can produce files for matching that include exit dates, goal and employment status for each student. Yes No

• State has established a procedure for collecting Social Security numbers, including how to deal with missing numbers. Yes No

• State has set a regular schedule for data submission from local programs and for data matching with external agencies. Yes No

5. States has provided to all local programs a copy of the assessment policy and an explanation of the policy. Yes No

6. State has provided to all local programs the written state policies, procedures and requirements for student follow-up and an explanation of the procedures. Yes No

7. The state has written definitions for all measures (including demographic measures and actual or proxy contact hours, if applicable), defined according to NRS requirements and has provided them to all programs. Yes No

8. The State has written policies on the use of proxy hour models to assign the proxy hours for distance education learners participation in adult education distance education programs. (Check NA and skip to the next item if proxy hours are not used)

NA Yes No

8a. If yes, please identify which model or models were used to assign proxy hours

___ Clock Time Model

___ Teacher Verification Model

___ Learner Mastery Model

Superior Quality

1. The state has a comprehensive data dictionary, which defines all measures on state student data forms and in the state data system, and has provided it with an explanation to all local programs. Yes No

2. State has standards or requirements for the percentage of students to be pre- and posttested. Yes No

2a. If yes, indicate the standards or requirements.

3. State has made available to local programs on a continuous basis additional technical assistance and resources on assessment, data collection and follow-up procedures (e.g., site visits, contact persons, manuals, online resources). Yes No

3a. If yes, briefly describe the assistance and how it is provided.

4. If state uses survey follow-up method for any measure, the state has taken steps (e.g, through data review, discussion with staff or observation) to verify that the survey is being conducted according to NRS guidelines (check NA and skip to the next item if survey is not used). NA Yes No

4a. If yes, briefly describe your verification procedures.

5. If state uses survey follow-up method, the state has provided written guidance or assistance on how to improve response rates to survey staff (check NA and skip to the next item, if survey is not used). NA Yes No

6. If state uses data matching, the state has written procedures on how to conduct data matching that comply with NRS guidelines (check NA and skip to the next item if data matching is not used). NA Yes No

7. State has procedures in place that verify whether proxy hours are calculated and assigned appropriately (check NA and skip to the next item if proxy hours are not used). NA Yes No

Exemplary Quality

1. State has a system for verifying that local programs are following state data policies and procedures through program reviews, auditing or a certification process. Yes No

1a. If yes, briefly describe your verification procedures.

2. State has conducted (or reviewed reports of) the validity, reliability and comparability studies of its assessments and other data collection instruments. Yes No

2a. If yes, briefly describe how you conducted these studies.

Data Collection and Verification

Acceptable Quality

1. The state has an electronic management information system (MIS), used by all programs, that has individual student records within a relational data base structure. The MIS incorporates NRS measures using common definitions and categories. Yes No

2. Database has error checking functions used by state and/or local programs (e.g., that identify out-of-range values and missing data). . Yes No

3. State has standardized forms (electronic or paper) for collecting student information (e.g., intake, attendance, goal setting) that include all NRS measures and have correct NRS definitions and categories. Yes No

4. All programs are required to use state student data forms. Yes No

5. State has provided to local programs guidelines or procedures for recording actual and, if applicable, proxy contact hours that conform to NRS requirements. Yes No

6. All or most local programs have staff with clear responsibility for data collection and data entry. Yes No

7. State staff checks data for errors after submission by local programs. Yes No

7a. If yes, explain error checking process, including what data are checked and how often.

Superior Quality

1. Programs and/or state at least quarterly data entry into MIS. Yes No

2. State staff reviews local data at least quarterly for errors, missing data, out-of-range values and anomalous data, and to identify program improvements and accomplishments and has a system to resolve them. Yes No

3. State has timely (e.g., quarterly) follow-up back to local programs to have them correct missing and erroneous data. Yes No

4. State has documented procedures for correcting errors and resolving missing data that programs use. Yes No

4a. If yes, briefly explain your data review and error correction system.

5. State provides additional technical assistance to local programs with poor data, as needed. Yes No

Exemplary Quality

1. State has a regular system for verifying (through software, onsite auditing, contact with local staff) that local programs are following state data collection procedures. Yes No

1a. If yes, briefly describe the methods used for verification, including use of the correct assessments and assessment forms, reporting of accurate score ranges for placement and for reporting advancement for accountability.

2. State verifies data have been corrected in state or local database after errors have been found. Yes No

3. State has procedures for regular contact with local staff on data issues to identify problems and provide assistance. Yes No

3a. If yes, specify procedures and type of contact.

4. If state uses survey follow-up method, state tracks survey response rates on at least a quarterly basis and takes corrective action if problems are identified (check NA and skip if survey is not used). NA Yes No

Data Analysis and Reporting

Acceptable Quality

1. The state MIS can produce NRS required reports for state, including federal NRS tables. Yes No

2. NRS tables are calculated accurately to include error checks and prevent double counting. Yes No

3. State staff (or designee) checks NRS reports for errors and missing data and obtains corrected data from local program reports. Yes No

4. The MIS is capable of reporting disaggregated data by subpopulation (e.g., student age, race, sex) and program (e.g., ABE, ESL, ASE, correctional education, distance education). Yes No

Superior Quality

1. State staff person familiar with the data, but not directly involved with collection and data entry, reviews NRS data reports for errors and accuracy. Yes No

2. State staff uses data for program management and improvement. Yes No

3a. If yes, provide at least one example of use of data for this purpose in the last year.

3. Local programs can access data reports that are useful for program management and improvement. Yes No

4a. If yes, briefly describe the usefulness of two reports produced by your system.

4. Local staff uses data for program management and improvement. Yes No

Exemplary Quality

1. State has a system of regular contact with local programs on data analysis issues and reporting needs to identify technical assistance needs. Yes No

2a. If yes, specify method and frequency of contact.

2. State has documented procedures for dealing with analysis problems and deviations. Yes No

3. State compares data among programs and with prior years’ data for discrepancies, reasonableness and to identify trends in good and bad performance. Yes No

4. State has procedures to verify that local reports accurately reflect data collected (e.g., through review of local program documentation, onsite auditing). Yes No

5a. If yes, describe the report verification process.

Staff Development

Acceptable Quality

1. Local programs and state staff have been provided training on general NRS requirements, including assessment policy and procedures, follow-up policies and goal setting procedures. Yes No

1a. If yes, briefly describe when the most recent training occurred, its duration and about what percent of local providers attended.

2. Local staff has received training on data collection procedures. Yes No

3. State and local staff have been trained on data entry into the state or local MIS. Yes No

4. Local staff has had training on how to produce and/or interpret reports produced by the MIS. Yes No

5. Training has been provided on conducting follow-up survey or data matching procedures, to state or local staff involved in survey or matching. Yes No

6. State provides at least one additional training annually to local programs on NRS issues, MIS data entry or data analysis issues. Yes No

6a. If yes, briefly describe when the most recent additional training occurred, its duration and about what percent of local providers attended. This training should not be the same as the one described above in item number 1.

7. State has trained staff on distance education policy and use of proxy hours, if estimated. Yes No

Superior Quality

1. There is planned, continuous training (at least one training annually) on data collection and NRS issues. Yes No

1a. If yes, briefly describe frequency, duration and content of trainings.

2. NRS training is planned and delivered based on needs of local staff and evaluations of previous trainings. Yes No

2a. If yes, briefly describe your needs assessment process.

3. State has ongoing technical support to local programs to improve data matching and/or survey follow-up procedures, such as collecting the data and setting goals. Yes No

3a. If yes, describe support and how it is provided.

Exemplary Quality

1. State has developed and is implementing a plan for ongoing staff development on NRS and data use issues to promote continuous improvement. Yes No

1a. If yes, briefly describe the plan.

2. State has a system for continuous training of local staff on NRS issues, data collection and data reporting through regularly scheduled training sessions or other resources. Yes No

3. State has timely intervention strategies to identify data problems as they occur and to provide training to programs to correct the problems. Yes No

3a. If yes, briefly describe the process.

A Project of the U.S. Department of Education

Data Quality Improvement Plan

The state must submit a quality improvement plan for each content area that does not meet all of the standards within the acceptable level. A separate plan must be completed for each content area. Optionally, the state may submit a plan for content areas that meet acceptable level standards, but not superior level. The plans should not exceed one page and include the following information.

1. Content area (e.g., Data Foundation and Structure, Staff Development) and specific standard(s) not met.

2. For each standard not met, describe your planned approach to implementing changes that will allow you to meet the standard.

3. Describe the barriers or problems you anticipate, if any, to implement these plans.

4. Describe any technical assistance you might need to implement these planned changes.

5. If you believe you will be unable to meet any standard please explain why.

A Project of the U.S. Department of Education

nrs data quality Checklist Certification

Note: The state director of adult education or head of the state administrative agency in which the federal adult education program resides must sign this certification.

Certification

I certify that to the best of my knowledge, the information contained in this document is true and correct and accurately reflects the state’s data collection policies and procedures for collecting and reporting data for the U.S. Department of Education’s National Reporting System for adult education.

Signature

Name and Title

Date

Seal

12