AEE 577: Evaluation in Agricultural and Extension Education

NC STATE UNIVERSITY

"Probably the most difficult and time-consuming task in designing and conducting any program evaluation is developing the instrument to collect the information.”

 Emmalou Van Tilburg Norland

(Controlling Error in evaluation Instruments, 1990)

Class X:  Designing Evaluation Instruments
 

The survey method is the most common data collection approach in evaluation. The survey designed to collect evaluation data is referred to as an evaluation instrument. Evaluation instruments can be designed to collect quantitative as well as qualitative data. This lesson will explore the steps and concepts used for designing evaluation instruments.

Instructional Objectives:

Upon completion of this lesson, students should be able to:

1. List the step by step procedures for developing quality evaluation instruments;

2. Describe the errors that must be controlled in evaluation instruments;

3. Develop different forms of questions to record outcomes such as change in knowledge, attitudes, skills, aspirations, and behaviors;

4. Write process evaluation questions;

5. Describe reliability and validity

6. Identify double barreled questions; and

7. Develop an evaluation instrument.

Required Instructional Activities

1. View the Power Point presentation. It provides an overview of designing tools for recording change of knowledge, attitudes, skills, aspirations, and behavior.

2. Use the NEFE evaluation tool kit to design an evaluation tool. This web resource is useful for designing evaluation tools for financial education programs. It can be used to design evaluation tools for other extension programs as well. There are two resources in this tool kit- a database for designing evaluation surveys and a manual for evaluation. The evaluation manual provides evaluation guidelines. Chapter IV of the manual provides detailed instructions to use the database.

3. Read "Using Surveys" (pages 257-291) in the course text, Handbook of Practical Program Evaluation by Wholey, Hatry, and Newcomer (2004).

4. Read the following articles on designing evaluation instruments:

Taylor-Powell, E.  (1998).  Questionnaire design: Asking questions with a purpose.  University of Wisconsin, Cooperative Extension.  Available: http://learningstore.uwex.edu/pdf/G3658-2.PDF

Diem, K. G. (2002).  A step-by-step guide to developing questionnaires and survey procedures for program evaluation & research.  Rutgers University, Rutgers Cooperative Extension. Available: http://www.rcre.rutgers.edu/pubs/publication.asp?pid=FS995.

Van Tillburg Norland, E. (1990).  Controlling error in evaluation instruments.  Journal of Extension, 28(2). Available: http://www.joe.org/joe/1990summer/tt2.html.

Suggested Instructional Activities

1. Summerhill, W. R., & Taylor, C.L. (1992). Writing questions for mail questionnaires. Gainesville, FL: University of Florida, Cooperative Extension Service.  Available: http://edis.ifas.ufl.edu/pdffiles/PD/PD02300.pdf.

2. Taylor, C.L. & Summerhill,  W.R. (1992). Formatting a mail questionnaire. Gainesville, FL: University of Florida, Cooperative Extension Service.  Available: http://edis.ifas.ufl.edu/pdffiles/PD/PD02400.pdf.

3. Taylor, C.L. & Summerhill,  W.R. (1992). Writing options for mail questionnaires. Gainesville, FL: University of Florida, Cooperative Extension Service.  Available: http://edis.ifas.ufl.edu/pdffiles/PD/PD02500.pdf.

4. Israel, G. D. (2000). Conducting a customer satisfaction survey. Gainesville, FL: University of Florida, Cooperative Extension Service.  Available: http://edis.ifas.ufl.edu/pdffiles/WC/WC03900.pdf.

Required Assignments

All items, unless otherwise noted, are due by 12:00 noon on Monday following the week of the class.

1. Post your questions or comments on the Web CT Vista discussion board and reply to at least two postings submitted by your classmates.

2. Reading Assignment # 8 from class X readings is due by March 23: Open this reading assignment, complete and e-mail as an attachment to the instructor (jay_jayaratne@ncsu.edu). Remember to type your name and save your completed assignment before you e-mail it.

3. Develop an evaluation instrument for your term group project.  Your term project evaluation instrument is due by March 25.

References

Diem, K. G. (2002).  A step-by-step guide to developing questionnaires and survey procedures for program evaluation & research.  Rutgers University, Rutgers Cooperative Extension. Available:http://www.rcre.rutgers.edu/pubs/publication.asp?pid=FS995.

Israel, G. D. (2000). Conducting a customer satisfaction survey. Gainesville, FL: University of Florida, Cooperative Extension Service.  Available: http://edis.ifas.ufl.edu/pdffiles/WC/WC03900.pdf.

Rossi, P. H, Freeman, H. E., & Lipsey, M. W. (1999). Evaluation: A systematic approach (6th edition).  Thousand Oaks: Sage Publications. (ISBN 0-7619-0893-5)

Summerhill, W. R., & Taylor, C.L. (1992). Writing questions for mail questionnaires. Gainesville, FL: University of Florida, Cooperative Extension Service.  Available: http://edis.ifas.ufl.edu/pdffiles/PD/PD02300.pdf.

Taylor-Powell, E.  (1998).  Questionnaire design: Asking questions with a purpose.  University of Wisconsin, Cooperative Extension.  Available: http://learningstore.uwex.edu/pdf/G3658-2.PDF

Taylor, C.L. & Summerhill,  W.R. (1992). Formatting a mail questionnaire. Gainesville, FL: University of Florida, Cooperative Extension Service.  Available: http://edis.ifas.ufl.edu/pdffiles/PD/PD02400.pdf.

Taylor, C.L. & Summerhill,  W.R. (1992). Writing options for mail questionnaires. Gainesville, FL: University of Florida, Cooperative Extension Service.  Available: http://edis.ifas.ufl.edu/pdffiles/PD/PD02500.pdf.

Van Tillburg Norland, E. (1990).  Controlling error in evaluation instruments.  Journal of Extension, 28(2). Available: http://www.joe.org/joe/1990summer/tt2.html.

Wholey, J. S., Hatry, H. P., and Newcomer, K. E. (2004). Handbook of practical program evaluation (2nd edition).  Jossey-Bass, San Francisco, California.