|
REPORTING EVALUATION RESULTS: 10 COMMON MYTHS
Richard A. Krueger University of Minnesota
A paper presented at the Annual Meeting of the American Evaluation Association Kansas City, Missouri November 1, 1986
Evaluation results are sometimes not used or even ignored because of inadequate or ineffective reporting. This paper examines the importance of reporting evaluation results within the Cooperative Extension Service environment. It has been an accepted notion that extension workers possess above average skills in oral and written communication. Unfortunately, many find themselves ill prepared and uncomfortable when requested to present evaluation findings. The value of the evaluation findings are minimized or lost altogether when results are ineffectively communicated.
Reporting evaluation results is an overlooked art. It is an art in the sense that it defies precise scientific dissection. The general principles of reporting are helpful, but still the individual must meld these precepts with individual skills, abilities and talents to arrive at the best possible report. Reporting has been largely ignored in the evaluation literature, or at least relegated to cursory treatment. Our attention as evaluators has often been placed on methodology, on statistical procedures, and on other matters of limited interest to lay people.
In the past three years, I have had the opportunity to work with scores of extension workers as they prepared their evaluation reports. In these experiences I have served as an active listener and at times the coach and advisor as they practiced their presentations. These experiences have resulted in several observations about the presentation of evaluation reports in an extension environment.
Extension staff tend to have several misperceptions or hold mistaken myths about evaluation reporting that interfere with the quality of the oral and written report. Before we examine these misperceptions, we must emphasize the importance of quality information to report. Quality reporting assumes that the evaluation was conducted in an acceptable manner, and that the analysis met with traditional criteria in the research/evaluation community. Skillful reporting is no substitute for inadequate design, improper analysis procedures, or fuzzy thinking about the evaluation process.
Myths of Reporting
Assumptions and previous experience with research oriented reports have caused difficulty in evaluation reporting. As I have worked with extension workers in the preparation of evaluation reports, I have encountered ten myths about evaluation reporting that limit the ability of evaluators to communicate. These myths relate both to written reports (Myths 1-5) and to oral reports (Myths 6-10).
Myth 1. One written report is enough
Those who prepare evaluation reports tend to assume that people prefer to learn about results in the same way the evaluator prefers. Evidence suggests that people differ on how they prefer to receive information. Individual differences and preferred learning styles have been well documented in educational literature.
The implication for evaluators is to prepare multiple reports using a variety of media. Thus far we have presented written, oral and recorded reports. The procedures we use are only limited by our resources and creativity.
A related point is the use of multiple written reports. Minnesota impact studies are typically written in a minimum of three different ways. The first is the unabridged evaluation report which is prepared in limited quantity. The second is the brief narrative abstract for use on the Narrative Accomplishment Reporting System, the computer data base for the Extension Service – United States Department of Agriculture. The third, and perhaps the most important, is a rather brief, popularized report intended for state legislators, county decision makers and other interested parties.
Myth 2. People read written reports
Our impact study reports are targeted primarily to state legislators and local county decision makers. In follow-up interviews with these individuals we have discovered that our evaluation reports are rarely read, and sometimes not even recognized when we show them a copy which they have previously received. It appears that the primary readers may be very few and are usually existing internal staff members, or extension evaluation staff from another state or the U.S. Dept. of Agriculture. Elected officials are deluged by enormous quantities of reports and one should not be surprised that our results, or results of any other agency, are put aside, filed or just thrown away.
In spite of this general lack of report reading, there remain several reasons for continuing to write evaluation reports. First, it is a disciplined effort that helps the evaluator arrange the findings, conclusions and recommendations in a logical sequence that can be subjected to peer review. This disciplined effort results in tighter logic, more precise statements, and an overall improvement of the report quality. Secondly, we suspect that some of the information from the reports is transmitted by word of mouth by “influentials”. Influentials might quote reports or cite findings and thereby exert considerable influence on others. In Minnesota we have purposely decided to share impact study findings first with the state citizen’s advisory council, thereby giving them the inside information on program results. At the county level this has occurred with local advisory groups or county extension committees.
Myth 3. Written evaluation reports are the same as research reports
We learn to write research reports as undergraduates and then refine and hone these skills in our graduate studies. Those engaged in evaluation efforts are typically well versed in research reports and consequently often use academic research reports as exemplars for evaluation reports. Unfortunately, these models from academic research have severe limitations when applied to program evaluation. I’d like to highlight several distinctions between research reporting and evaluation reporting.
First, we encourage the authors of evaluation reports to write for specific people – as opposed to the evaluation or research community. The specific people are often elected officials, local influentials, or lay people with a specific interest in the subject. As a result the writing style is less formal; shorter words are preferred to more complex words, technical jargon is avoided, and examples of concepts, quotations, or illustrations are encouraged.
Second, the evaluation report begins with an executive summary typically limited to two pages or less. The research report by contrast usually includes the summary at the end.
Third, lengthy explanations of procedures or methodology are avoided in the popularized evaluation reports. The overall emphasis of the evaluation report is to communicate the findings, conclusions and recommendations as directly and precisely as possible. Research reports typically provide more detailed treatment of procedures, in part to allow for replication by other researchers.
We have developed an outline for written evaluation reports which build on the concepts of Morris and Fitz-Gibbon in their excellent publication: How to Present an Evaluation Report. Our outline, included in the appendix, has been helpful in establishing standardized procedures for evaluation reports.
Myth 4. Complex statistical analysis and big words impress the audience
When writing for lay audiences the complex descriptions of analysis and academic jargon actually inhibit understanding. This is especially a problem for extension specialists who may wish to demonstrate their academic acumen to their professional colleagues. Complex statistical procedures are perfectly acceptable in the evaluation, but they must be explained in a meaningful way to those not acquainted with research procedures. When editing the written reports we typically look for several types of inappropriate words. They might include specialized jargon commonly used within a specific discipline, “in house” extension words, or just big words that have vague or multiple meanings.
Myth 5. Describing limitations weakens the report
We have recommended that all evaluation reports prepared by staff members of the Minnesota Extension Service contain an identified section on “Limitations and Alternative Explanations”. This section actually enhances the credibility of the report, providing that the limitations are conscientiously prepared. In this section the evaluator might cite concerns relating to sample size, response rate, or other factors which would limit the application of the findings to other situations or geographic areas. At times the findings can yield different interpretations of program results, and these alternative explanations are contained in this section of the written report.
Myth 6. Oral reports have same sequence as written reports
The outline for a written report does not travel well in oral reporting. Often we make the assumption that a report is a report, whether it be oral or written and that the sequence of information presented should be consistent. Oral reporting is different and it requires some special forethought and preparation. Quality oral reporting is closer to an art than a science.
The first few moments of the oral report are critical. Within these first few moments the evaluator should highlight several key factors. For example: Why was the evaluation needed? What do we know now that we didn’t know before? or How can these findings be used? It is important to quickly engage the audience, to involve them in the report, to hook them in to the evaluation and explain clearly why the evaluation was important.
When planning for the oral report it is helpful to give consideration to the “ho-hum syndrome”, a typical reaction of elected officials. “Ho-hum” is best described by the questions going through the minds of the audience, such as: Do we really need this study? Don’t we know this already? or Shouldn’t this staff member be doing something really important instead of just evaluating? To us the results might seem enormously important with far reaching implications, but to a busy elected official it might sound like hair-splitting and avoidance of real work. Much of what we discover in program evaluation efforts does tend to sound like common sense, and this tendency needs to be defused in the oral presentation. Often the best procedure is to address it head-on by saying, “This study is of importance to us because .”
Myth 7. Build to the most important point – save it for last
Some communications experts have recommended that the most important points be presented at the end of an oral presentation – that lesser points build toward the most critical point. This recommendation is helpful in a number of presentation environments, but it does not work well in evaluation reporting. Most evaluation reporting occurs in environments with time restriction, decision makers with limited patience, and regular interruptions. In these situations brevity and conciseness are valued, and the most important findings are therefore better placed at the top of the list of results.
A related issue concerns the number of points included and the phrasing of those points. We have recommended that evaluators present oral reports with less than seven points. The basis of this recommendation stems from studies in cognitive psychology which suggest that five to seven items are the short term memory capacity of most people. In addition we have encouraged the use of short, active phrases to describe points as opposed to complete sentences. These brief phrases are designed to do two things: to convey the important concept and also to be easily remembered.
Myth 8. 15 minute report means a 15 minute presentation
Those receiving an oral evaluation report usually wish to discuss findings, respond to the results, or ask questions. The most successful oral reports have allocated only one-third to one-half of the time for the presentation and the remainder is spent in follow-up discussion. Therefore the 15 minute report includes a 5 minute presentation and 10 minutes for questions, clarifications and discussion of future action.
Myth 9. The audience knows why they are getting the report
Have you ever given an oral report to a local elected body and when finished the group just looked at each other for a few awkward moments. This uncomfortable silence is often followed by some type of action typical of elected bodies. Someone might move that the report be approved or accepted. Then they can move on to really important matters. Often the group receiving the oral report does not know why they are receiving the briefing simply because we never told them why it was being presented. At the end of the report the presenter should indicate what action is recommended or why the report was presented such as: to provide a briefing, to form a study committee, to continue discussion at a later time, to seek funds to implement the findings, to approve a new course of action, etc. It is dangerous to assume that the audience will know what to do with the report.
Myth 10. Everything should be reported
There is a tendency in oral presentations to report everything that was discovered, and unfortunately much of the information may be of limited or no concern to the audience. Typically we learn a lot from our evaluation efforts, and some of this new information is of immediate usefulness, but other information might be of more passing interest with limited or not immediate utility. Oral reports must be brief and therefore only the most critical information should be presented. Know your audience and present findings that are of greatest concern or usefulness to the decision makers.
Summary
In summary, quality evaluation reports begin with quality evaluations, but not enough attention has been spent on communicating the results of our efforts. We have made assumptions that need to be reexamined in light of experience and evidence. One report is not sufficient, written reports may be overvalued, evaluation reports do differ from research reports, complexity can be confusing, citing limitations strengthens the report, oral reports are structured differently from written reports, the most important point is presented first, audiences may need suggestions of how to respond, the evaluator should clearly indicate why the report is being presented, and oral reports should focus on critical information.
Evaluation Reporting Outline
1. Cover Page
The front cover should include: Title of the program and its location Name of evaluator and names of task force members Names of people to whom the evaluation report is to be submitted Period covered by the report Date the report is submitted
2. Summary
Begin with a brief, well written summary describing why the evaluation was conducted and listing its major conclusions and recommendations. It should be no longer than 1-2 pages. It should be able to stand alone. Although this section is placed first, it is often the part that you write last!
3. Table of Contents
This section is optional and need not be included when the entire report is relatively brief. The table of contents provides the reader with information on how the report is organized, where various parts are located, and how to access areas of particular interest.
4. Background Information on the Program
Describe the problem, program or activity being evaluated. You might include information on: the origin of the program, the goals or objectives of the program, a description of the clientele and others involved in the program.
5. Description of the Evaluation Study
Describe why the evaluation was conducted and clearly state the evaluation question(s) you sought to answer. Describe the procedures used to collect evidence and explain why those procedures were used. Where did the evidence come from? Who participated? Describe sampling procedures, response rate, pilot testing of instruments, and how instruments were developed.
6. Results
Use qualitative or quantitative procedures to place results into categories, make comparisons where appropriate and form interpretations. Tables, charts and statistical procedures can help describe the results. Draw conclusions. How do the results obtained compare to what was intended or expected or to a control/comparison group?
7. Limitations and Alternative Explanations
This section could be placed within the Results category if it is brief. Limitations refer to aspects of the evaluation study that limit your ability to transfer findings to other areas, limitations due to use of procedures that prevent conclusive statements about the programs, and limitations relating to response rates or sample sizes. Are there alternative explanations or interpretations of program results that differ from what we described in the Results section? Lengthy alternative explanations might be included in the appendix.
8. Implications – Next Steps
This section should describe what can or should be done differently in the program based on the results of the evaluation.
9. Appendix
The appendix is the appropriate place for: A. Examples of instruments used to collect evidence, such as questionnaires, survey forms, observations sheets, etc. B. Data summaries C. Alternative explanations of survey D. Lesson plans (if evaluation is of a particular lesson). E. NARS Reports F. Background information that might further clarify the evaluation.
References
Brinkerhoff, R.O., ct.al. Program Evaluation Sourcebook. Boston: Kluwer-Nijhoff, 1983. Chapter 5.
Morris, Lynn Lyons and Carol Taylor Fitz-Gibbon. How to Present an Evaluation Report. Beverly Hills: Sage, 1978. Krueger 1986
|