What Kinds of Ethical Problems Do Evaluators Encounter?
Studies of practicing evaluators reveal the types of ethical challenges that evalua- tors face. Morris and Cohn (1993) surveyed members of the American Evaluation Association and found that nearly two-thirds of the evaluators had encountered major ethical challenges in their evaluation work. Their analysis of the types of ethical violations that members encountered showed these types of problems:
A. Challenges in the contracting phase: • Stakeholder has already decided what the findings “should be” or plans to
use the findings in an ethically questionable fashion.
80 Part I • Introduction to Evaluation
• Stakeholder declares certain research questions off-limits in the evalua- tion, despite their substantive relevance.
• Legitimate stakeholders are omitted from the planning process. B. Ethical concerns regarding confidentiality or disclosure agreements:
• Disputes or uncertainties concerning ownership/distribution of the final report, raw data, etc.
• Although not pressured by stakeholders to violate confidentiality, the evaluator is concerned that reporting certain findings could represent such a violation.
• Evaluator is pressured by stakeholder to violate confidentiality. C. Challenges in presenting findings:
• Evaluator is pressured by stakeholders to alter presentation of findings. • Evaluator is reluctant to present full findings for unspecified reasons. • Evaluator has discovered behavior that is illegal, unethical, dangerous, etc. • Evaluator is unsure of his or her ability to be objective or fair in presenting
findings. D. Ethical concerns after the report is complete concerning misinterpretation
or misuse: • Findings are suppressed or ignored by the stakeholder. • Unspecified misuse by the stakeholder. • Findings are used to punish someone (the evaluator or someone else). • Findings are deliberately modified by the stakeholder prior to release. • Findings are misinterpreted by the stakeholder (Morris & Cohn, 1993,
pp. 630–632).
Morris and Cohn’s study remains one of the few to empirically examine the ethical challenges that evaluators face in their work. The most frequent category of problems occurred in preparing results: almost two-thirds of the evaluators reported being pressured by stakeholders to alter results. Morris and Cohn draw several interesting conclusions from their study. First, their content analysis of responses revealed that ethical problems “can, and do, arise in every stage of eval- uation” (1993, p. 639). Although respondents reported problems at every stage of the evaluation, the most frequently cited problems occurred at the final stages of the evaluation, in presenting findings. These ethical problems generally arise from pressures from stakeholders, typically the client, concerning the product of the evaluation. In other words, stakeholders are less likely to apply pressure as the study is being carried out than with the final product, the evaluation findings, and the report. In fact, clients presumably value the scientific and objective nature of the work they have hired the evaluator to conduct. But, their concerns emerge with the product itself when the results are surprising or disagreeable. When clients or other stakeholders argue with the evaluator over the interpretation of the results or the presentation of findings, the evaluator may be surprised, hav- ing conceptualized his or her role as an independent, objective evaluator. Thus, Morris and Cohn note, the stakeholders’ pressures, as seen by the evaluator, “undermine the mission of scientific inquiry, which is to seek the truth and
Chapter 3 • Political, Interpersonal, and Ethical Issues in Evaluation 81
communicate it” and the evaluators “feel pressured to compromise their role as scientists” (1993, p. 639). These conflicts reveal the “clash of cultures” described by Eleanor Chelimsky and discussed at the beginning of this chapter. That is, stake- holders in this political context are competing for resources, power, and leader- ship. They see the evaluation findings as a tool that they can use to their benefit in that competition. The evaluation is valued because of its perceived objectivity. But, when the findings clash with their needs in the competition, the political context becomes more important to the stakeholder than the continued objectiv- ity or independence of the evaluator’s conclusions.
Faced with such ethical conflicts, the evaluator must take a stand to protect the credibility of this evaluation and future ones. The situation is not an easy one. It is relatively rare for stakeholders to ask evaluators to actually change data. And, to bring about use, good evaluators generally seek the input of clients and other stakeholders on the interpretation of results and presentation of findings in draft reports. So, when the client gives feedback, suggesting changes, evaluators may interpret these “suggestions” differently. (What is the nature of the suggestion? Does it result in a major difference in the interpretation? How strongly does the client ask for or even demand changes?) So the request for changes must be inter- preted by the evaluator. Of course, in some cases, the ethical challenge would be quite clear: The client demands that the evaluator change major conclusions con- cerning the quality of the program. In other cases, the client may be asking for what the client perceives as editing changes, but the evaluator sees as watering down the clarity or strength of the judgments made. How does the evaluator handle this more ambiguous ethical challenge? Dealing with the first situation, in which major conclusions on the quality of the program are demanded, the obvious, ethical chal- lenge requires courage and integrity on the part of the evaluator to maintain the validity of the findings. Dealing with the second ethical challenge certainly may, in the end, require courage and integrity, but may initially require careful thought and reflection concerning the intentions of the client’s editing suggestions and the ownership of the report, its wording, and its conclusions. Finally, both situations require the evaluator to recognize that an ethical challenge has occurred.
Although the Morris and Cohn study reveals much of interest concerning the types of ethical conflicts that evaluators actually encounter, they are also concerned that one-third of their sample reported they had not encountered any ethical con- flicts in their evaluation work. Their concern, rightly so, is that these evaluators are not just operating in safer environments, but, instead, are not recognizing ethical conflicts or challenges when they arise. As Morris and Cohn conclude, “The subjec- tive notions of ethicality held by many unchallenged group members [those not reporting an ethical challenge] differ in systematic ways from those held by members of the challenged group” (p. 635). Since their study was concerned with describing the ethical problems that evaluators encounter, they were unable to explore the rea- sons for these different notions of ethical behavior. However, they recommend, and we concur, that the differences illustrate the need for education and training for evaluators to discuss and explore the ethical challenges they may encounter—how to recognize and interpret them, and, ultimately, how to deal with them.
82 Part I • Introduction to Evaluation
One of the few other studies on ethical behavior among evaluators took a qual- itative approach by asking a smaller number of evaluators to discuss how they dealt with ethical issues in their work (Honea, 1992). Honea found that these evaluators seldom discussed ethics or values in their work lives. She identified four factors that seemed to inhibit such discussions. Specifically, her interviewees perceived that:
1. They were being ethical if they were following the model of “objective scien- tist,” and lapses in objectivity were viewed as less an ethical than a method- ological concern;
2. Participants in evaluation always behave ethically, so discussion of ethics is unnecessary;
3. Being a member of an evaluation team and engaging in team deliberations prevents unethical behavior from occurring;
4. Neither evaluators nor others involved in the evaluation have the time to confront or discuss ethical issues.
These studies suggest that more attention should be given to ethical issues in educating and training evaluators. In the next sections we discuss the professional codes that can be helpful to evaluators in raising their awareness of ethical obli- gations and in communicating professional obligations to stakeholders.