Reflecting on Sources of Bias and Conflicts of Interest
Several of the Program Evaluation Standards (U.1, U.4, P.6, A.8) and Guiding Principles (C.3 and C.4) are concerned with the importance of evaluations being honest and impartial, avoiding conflicts of interest and conducting evaluations with integrity. Yet, as research has shown, many evaluators do not believe they have encountered ethical problems and think that because they are following accepted social science methods and the model of the “objective scientist” they are, of course, behaving ethically (Honea, 1992; Morris & Cohn, 1993). In this section, we would like to dis- cuss potential biases and conflicts of interest that evaluators must consider carefully.
Chapter 3 • Political, Interpersonal, and Ethical Issues in Evaluation 95
First, we should acknowledge that the possibility of human beings rendering completely unbiased judgments is very slight. In fact, it is ironic that some evalu- ators actually could be more susceptible to bias, simply because they believe that by using social science methodologies to draw conclusions, they are objective and unbiased. But, Carol Weiss, one of the founders of evaluation notes, “You never start from scratch. We pick up the ideas that are congenial to our own perspective. Therefore, people pick up this thought or that interpretation of a research report that fits with what they know or what they want to do” (2006, p. 480).
Evaluators and those involved in evaluation should carefully reflect on their biases. By becoming aware of those biases, one can consider and perhaps counteract some of their influence on evaluation. The ethical evaluator recog- nizes that evaluation practice consists of making choices—choices about evalu- ation purposes and questions, about which stakeholders to involve and which designs and data collection strategies to use, about ways to analyze data and to interpret the results. Note, for example, that the raters at Moody’s or Fitch’s who were researching and assigning ratings to bonds were making choices, too— about what information was important and what was not, about what types of business and investment strategies were fiscally sound. Their ratings involved much more than simply adding together some numbers. Swayed by the tenor of the times, changing business practices, the interests of their rating company and those who paid them, their ratings were influenced in undesirable ways. Choices are, by nature, subjective. Evaluators increasingly realize that bias—inadvertent or conscious—can intrude subtly into nearly every choice they make, from selecting an evaluation approach to writing a report. To avoid the faulty findings of bond analysts, evaluators must think more carefully about the potential sources of bias and conflicts of interest that can occur in each evaluation they are conducting.
It is worth noting that when asked to describe ethical problems they have encountered, evaluators tend to describe problems presented by stakeholders (Morris & Cohn, 1993). As Morris and Cohn themselves note, it may be more difficult for evaluators to recognize or report ethical problems that were of their own doing. The only ethical problem they found that appeared to originate with the evaluator was the concern about their ability to be objective or fair in pre- senting findings. Recognition of this particular problem, though, is a major first step. It suggests that, even when encountering many ethical problems presented by client or stakeholder pressure, some evaluators remain conscious of how their own biases can interfere with the accurate presentation of results. (Of course, stakeholder pressure and concern with being objective and fair in presenting findings can overlap. When faced with strong pressure from a client, it may be difficult not to become biased against that client and overreact and become less fair and objective in the opposite direction. This might result in reporting or emphasizing problems either in retaliation or to show that you are objective, rather than maintaining a balanced view. It can be difficult to see things from the perspective of someone who has behaved inappropriately toward you, yet that is what the evaluator must do in order to consider all sides.)
96 Part I • Introduction to Evaluation
Guidance from Ethical Codes and Standards. The Joint Committee Standards and the AEA Guiding Principles can often serve as a good first step to raising awareness and considering potential problem areas. So, let us review a few of the Standards and Principles that are relevant to the issues of bias and con- flict of interest. Guiding Principle C concerns integrity and honesty. Principles C.2 and C.4 directly address expectations concerning values, interests, and relationships:
C.2 Before accepting an evaluation assignment, evaluators should disclose any roles or relationships they have that might pose a conflict of interest (or apparent conflict of interest) with their role as an evaluator. If they proceed with the evaluation, the conflict(s) should be clearly articulated in reports of the evaluation results. . . .
C.4 Evaluators should be explicit about their own, their clients’ and other stakeholders’ interests and values concerning the conduct and outcomes of an evaluation (American Evaluation Association, 2004, Section C Integrity/Honesty).