AEE 577: Evaluation in Agricultural and Extension Education

NC STATE UNIVERSITY

"A wide variety of techniques may be used singly and in combination to gather data.".... "As in all aspects of evaluation, the particular approaches used must take into account the resources available and the expertise of the evaluator. While 'high technology' monitoring and sophisticated quantitative analytical procedures are often the method of choice, in many cases relatively simple and economical data collection activities are all that is necessary.”

 Peter H. Rossi and Howard E. Freeman (1993, p.197)

Class VIII:  Methods (Techniques) of Data Collection

Evaluation data collecting methods can be categorized into two broad groups called quantitative methods and qualitative methods. Quantitative methods collect numeric data while qualitative methods collect descriptive data or information. Numeric data are called hard data and descriptive data are called soft data. Generally, decision makers like hard data. Sometimes numbers are not enough to describe the activities of a program. When you encounter this situation, you need to explore the evaluation possibilities offered by qualitative methods. Quantitative methods and qualitative methods have distinctive strengths and weaknesses. Therefore, it is advisable to use both methods to complement each other method. The application of qualitative and quantitative methods together is called mix method.

Classes VIII and IX will be spent for methods of data collection. You will learn Observation and Agency Record methods in this class.

Instructional Objectives:

Upon completion of this lesson, students should be able to:

1. Discuss the advantages, disadvantages and limitations to using trained observer ratings.

2. Describe what is meant by agency records.

3. Discuss potential problems in using agency records and how to overcome those problems.

4. Discuss when observation is useful in evaluating programs.

5. Describe the difference between structured and unstructured observations.

6. Develop an instrument to record direct observations.

7. Describe qualitative methods.

8. Discuss ways qualitative methods can be used in Extension.

Required Instructional Activities

1. For a brief overview of the various data collection methods presented in the text, read "Practical Data Collection Procedures" (pages 205-210) in the course text, Handbook of Practical Program Evaluation by Wholey, Hatry, and Newcomer (2004).

2. Read selected pages 215-220 and 252-253 from "Trained Observer Ratings" in the course text, Handbook of Practical Program Evaluation by Wholey, Hatry, and Newcomer (2004).  If you are interested in learning more details about trained observer ratings, the chapter will make a great resource.

3. Information needed to evaluate a program may have already been collected, read "Using Agency Records" (pages 396-410) in the course text, Handbook of Practical Program Evaluation by Wholey, Hatry, and Newcomer (2004), to see how you might use existing records.

4. View the Power Point presentation. It is the summary of class materials.

5. Review Michael Quinn Patton's (2003) Qualitative Evaluation Checklis.

6. Read Ellen Taylor-Powell and Sara Steele's article on  Collecting evaluation data: Direct observation.

Suggested Instructional Activities

1. You may be interested in the following articles:  These will not be on the test, but for your own use as resource materials.

   "Using Expert Judgment" (pages 292-309) in the course text, Handbook of Practical Program Evaluation by Wholey, Hatry, and Newcomer (2004).

   Gamon, J. A.. (1991). The delphi-An evaluation tool. Journal of Extension, 29(4). Available:  http://www.joe.org/joe/1991winter/tt5.html.

   Van Tilburg, E. (1987). Evaluation with a new twist . Journal of Extension, 25(4). Available:  http://www.joe.org/joe/1987winter/a1.html.

   Smith, M. F., & Lincoln, Y. S. (1984). Another kind of evaluation. Journal of Extension, 22(6). Available: http://www.joe.org/joe/1984november/a1.html.  Discusses qualitative research and applications to Extension.

Required Assignments

All items, unless otherwise noted, are due by 12:00 noon on Monday following the week of the class.

1. Post your questions or comments on the Web CT Vista discussion board and reply to at least two postings submitted by your classmates.

2. Reading Assignment # 6 from class VII & VIII is due by March 11: Open this reading assignment, complete and e-mail as an attachment to the instructor (jay_jayaratne@ncsu.edu). Remember to type your name and save your completed assignment before you e-mail it.

3. Complete this observation scale development exercise. This learning activity is not a grading assignment and you don't need to submit it.

References

Gamon, J. A.. (1991). The delphi-An evaluation tool. Journal of Extension, 29(4). Available:  http://www.joe.org/joe/1991winter/tt5.html.

Patton. M. Q. (2003). Quaitative evaluation checklist. Avaliable at: http://www.wmich.edu/evalctr/checklists/qec.pdf

Rossi, P. H, Freeman, H. E., & Lipsey, M. W. (1999). Evaluation: A systematic approach (6th edition).  Thousand Oaks: Sage Publications.

Smith, M. F., & Lincoln, Y. S. (1984). Another kind of evaluation. Journal of Extension, 22(6).  Available: http://www.joe.org/joe/1984november/a1.html.

Taylor-Powell, E., & Steele, S.  (1996).  Collecting evaluation data: Direct observation.  University of Wisconsin, Cooperative Extension.  Available: http://learningstore.uwex.edu/pdf/G3658-5.PDF

Van Tilburg, E. (1987). Evaluation with a new twist . Journal of Extension, 25(4). Available:  http://www.joe.org/joe/1987winter/a1.html.

Wholey, J. S., Hatry, H. P., and Newcomer, K. E. (2004). Handbook of practical program evaluation (2nd edition).  Jossey-Bass, San Francisco, California.