Lesson Eleven

Developing Evaluation tools for Accountability

Introduction:

Curriculum developers often create tools to be used to evaluate a student's mastery of the curriculum objectives. Often these are used in curriculum resources such as text, workbooks or as accountability tools. Teachers can use these evaluation tools for formative or summative purposes. Formative evaluations (often called interim or informal assessments) give teachers the information they need to know in order to adjust their instruction as they are teaching the curriculum. Summative evaluations are given to assign students grades or to assess their final mastery of an objective. End of course tests are summative evaluations.

In North Carolina, Workforce Development has created and adopted the VOCATS Instructional Management System. This system includes a blueprint, a curriculum guide, and test item banks that can be used for classroom assessment (formative) and accountability assessment (summative). The classroom assessment items are called unsecured item banks and the accountability assessments are called secured item banks. Teachers have access to the unsecured banks but do not have access to the secured banks. At the end of the school year, students in Workforce Development Education classes are given an end of course multiple choice test by a proctor. The test items come from the secured banks. Their test scores are reported to the North Carolina Department of Public Instruction. Many school systems count the student's test score as part of their final grade. The North Carolina Department of Public Instruction reports the scores to meet standards from the federal government.

Regardless of the reporting system, it is important that evaluation tools measure what they are supposed to measure. The test item should be written to match the domain of learning and level of the objective. In this lesson we will review good test writing techniques to help the curriculum developer write well-designed questions. We will also briefly define reliability of test scores and validity measurements of test items.We will explore item statistics which can help you as a teacher determine whether to keep, revise, or throw out a question.

 

Objectives:

  • Explain the different reasons we should evaluate students.

  • Choose the most appropriate tool for formative and summative evaluation.

  • Examine the use of evaluation tools for accountability purposes.

  • Differentiate between reliability and validity measurements of test scores and items.

  • Write questions at the level of behavior stated by an objective in a curriculum framework (blueprint).

  • Explore the test item development process used by the North Carolina Department of Public Instruction to create test item banks.

Readings:

Powerpoint Presentations:

Download this week's presentation:

Lesson 11 presentation

Examine the North Carolina VOCATS system test item development process and practice evaluating questions in a PowerPoint prepared by NCDPI to work with validation teams.

 

Assignments:

  • This week you should create a multiple choice question for each objective you wrote in lesson nine. You should create 20 questions (if you write a question for each objective). You should follow the North Carolina Department Of Public Instruction standards and hints for writing good questions in this week's lesson. Your test questions should match the domain of learning (cognitive, affective or performance) and the level of performance (ex. C1 and etc.). Please submit on Moodle.

  • The final project for this class will be due by November 30th and presented either December 1st or December 3rd. The final product consists of a PowerPoint and presentation that includes the four phases of your project completed in earlier classes. The rubric for the content requirements and the rubric for the presentation of the final project should be used as your development guide for this presentation. When developing this presentation keep in mind that your targeted audience is a county school board or county extension advisory board that is voting on whether or not to allow you to offer the course. (See the rubric for more detail.) Please remember the presentation you submit on November 30th should be the same one you present on December 1st or 3rd and should run no more than 20 minutes.

  • The final exam will be open book/ short answer/ multiple choice and will be available in Moodle November 30th - December 11th and will only cover the second half of the course.

Plugins to third party applications found in this lesson such as Quicktime, Powerpoint, Word, Adobe, etc can be downloaded from the AEE 529 course syllabus.