Interpersonal Relationships and Bias.
It is apparent to even the casual observer that individuals’ feelings toward one another can color their judgments, not only about each other but about practically anything with which the other person is perceived to be associated. Hence, we have legal restrictions on testimony about one’s spouse and anti-nepotism policies that prohibit individuals from being in positions where they would need to make decisions about the salary, promotion, or job security of a family member. Similarly, evaluators should avoid evaluating programs that a close friend or family member is concerned with, whether as a policymaker, a manager, or a person delivering the program. The apparent conflict of interest would be too strong even if the evaluator were able to overcome the bias the interpersonal relationship introduced.
Internal evaluators, except in the largest organizations, are almost inevitably evaluating programs that are staffed by someone they know. Therefore, internal evaluators need to think carefully about how to define their role in such settings. Even if the purpose of the evaluation is formative or for organizational learning, the evaluator needs to be prepared to give negative feedback. To achieve change, that feedback may be given in a way that is clear but palatable. Nevertheless, evaluators should be alert to examining how their relationships with those who operate or manage the program can influence the choices and decisions made. Such relation- ships can affect many elements of the evaluation, from the questions the evaluation addresses to the ways in which results are interpreted and presented. As an evalua- tor, you are hired or assigned to provide an independent, impartial judgment, and concerns about personal relationships should not interfere with the evaluator’s responsibility.
As we discussed earlier in this chapter, however, evaluators have a respon- sibility to develop some type of relationship with the client and stakeholders con- cerned with the evaluation. They must be able to communicate with them effectively so they can understand their needs and provide information in a way that meets those needs. Evaluators who are entirely new to the setting of the evaluation should spend time observing the program, meeting with clients and stakeholders, and developing relationships. These relationships are intended to help the evaluation to succeed—to reduce mistrust, to improve understanding, and so
100 Part I • Introduction to Evaluation
forth—but these relationships also introduce bias. Evaluators are likely to feel more comfortable with people whose values and beliefs are like their own, who support the evaluation and who are open to its methods and interested in its re- sults. At the same time, evaluators learn that some people concerned with the evaluation are more difficult. They are suspicious, accusatory, demanding, inflexible, or behave in any number of ways that are frustrating to the evaluator. These relationships, good and bad, influence the evaluator’s behavior. Is the eval- uator prepared to give tough, negative results—ones the evaluator knows they won’t like—to people with whom he has established rapport? To those who are helpful to the continuation of the study? These are tough issues, but evaluators must be prepared to deal with them, to prepare audiences for difficult results, and to prepare themselves for delivering them.
We find it is useful to clarify and demonstrate one’s role at the beginning of the evaluation, during the planning phase. Evaluators do not need to be the tough guy but they do need to be willing to ask tough questions and provide difficult feedback. In a later chapter, we will discuss using logic models during the early stages of the evaluation, as a method to help the evaluator understand the program. This is often a useful time to ask probing or tough questions such as, “Now, why is it that you think that this activity will lead to X change? Some of the research I have read doesn’t support that,” or “Which of your objectives do you think you are probably not achieving?” You may choose other ques- tions, but our point is that at the beginning of the study—not waiting until the end—you should start to define your role as someone who is interested in them and their program, but is also curious, objective, and questioning. This persona or manner then becomes part of your interpersonal relationship with others in the program.
Financial Relationships and Bias. Unfortunately, financial considerations are a source of bias in evaluation just as they were in the rating of bonds by Moody’s discussed earlier in this chapter. We doubt there are many instances of evaluators being bribed to sway an evaluation one way or another, but financial pressures are rarely so obvious and direct. To illustrate how thorny this situation can be, let us describe an actual case.
An evaluator of our acquaintance—we’ll call her Diane—was employed by a U.S. government-supported research center whose mission was to develop and test exemplary programs and practices for schools. Assigned to direct the center’s evaluation unit, in due time Diane completed an evaluation of a center program designed to improve secondary school students’ mathematics performance and attitudes toward math (AMP). The AMP program was expensive. Congress had invested more than $1 million in its development and, although Diane found that students liked the program, there wasn’t a shred of evidence to suggest that it had any impact on their performance. Troubled by the implications of reporting such information to the funding agency through which Congress had initiated the program, Diane finally worded her draft report to convey that the evaluation was
Chapter 3 • Political, Interpersonal, and Ethical Issues in Evaluation 101
to blame for AMP’s failure to produce evidence of success. The summary of her re- port read as follows (italics ours).8
Summary. The results of this study indicate that the Accelerated Mathematics Program (AMP) was somewhat effective in developing positive attitudes toward mathematics, in the sense that students tended to like the AMP materials. The study supplied no evidence, however, from which either long- or short-term student performance changes in mathematics ability can be inferred. The results do not necessarily indicate that AMP was not effective in promoting change in math performance, but that a variety of shortcomings and limitations of the evaluation design did not allow for the identification and measurement of these changes.
And how did the funding agency respond to this obvious effort to soften the bad news? Their reaction to the draft report came in a letter, which is reprinted here.
Dear Diane:
Thank you for the three draft copies of the AMP Impact study. I look forward to the final report.
I hope that our future efforts will be structured so that statements such as those in the “summary” will not have to be made. Instead, I hope that we will be able to say something positive in the final report about changes in important performances. I have heard so many good things about AMP that I am disheartened by the lack of evidence that it has short-term performance effectiveness and that I cannot therefore argue for its potential for long-term effectiveness.
The issue here is straightforward. The best argument for funding centers such as yours that I can make internally here in the Department and externally with the Congress is that our products lead to measurable changes for good in American schools. Re- gardless of the positive “feelings” I get about AMP, it appears we cannot justify all the effort in terms of performance criteria, as per your draft report. That is a drawback, but one which I think we can overcome in future efforts, hopefully in your final report.
Sincerely,
Lawrence T. Donaldson Chief Administrator
The message is blatantly clear. Diane better find something positive to prove AMP and its cohort programs are worth the investment, or funding could be with- drawn, the program would fold, and Diane herself would be looking for other employment. It would take a robust soul indeed not to feel some ethical strain in such a situation, especially when her salary comes directly from the threatened program!
8The names, organizations, and titles in this summary and the following letter have been changed to provide anonymity, but the essential content has not been altered and is reproduced here verbatim.
102 Part I • Introduction to Evaluation
Fortunately, though Diane equivocated at first, this story eventually had a happy ending. The final report told the true story, and Diane was able to assume the role of evaluator (with a clear conscience) on the development staff for another program at the same center.
Even when the evaluator is external to the agency whose programs or prod- ucts are being evaluated, financial dependence can be a potential source of bias. Consider, for example, the delicate balance that must be maintained by external evaluation consultants or firms who are inevitably dependent on repeat business. Scriven (1993) points out this potential source of bias succinctly: “. . . one key economic insight about evaluation contracting is this: No one ever got rich from one evaluation contract” (p. 84). The possibility of future evaluation contracts or consulting depends on how well the client likes the most recent evaluation com- pleted by the evaluator. No problem here if the client has a penchant for the truth, even if it might reflect negatively on the program. But what if the client goes rigid at the first hint of criticism? Developing formal agreements as indicated in the Standards can provide some assurance, but evaluators should always think carefully about financial relationships and recognize that their long-term reputa- tion as an independent, impartial evaluator is critical to their sustainability.
Organizational Relationships and Bias. Organizational relationships may be of greater concern to evaluators than immediate financial gain. The relationship between evaluators and the programs they evaluate can determine not only their present financial welfare but their future employment as well. Further, an orga- nization may exert great (or total) control over the evaluator’s other perquisites: such things as office space; access to resources, facilities, and record keeping systems; even the convenience of available parking space. The way the organiza- tion exercises this control to make the evaluator’s life easier or more difficult can certainly cause problems with bias.
To make this point, we present in Table 3.1 eight possible organizational rela- tionships between evaluators and the program being evaluated. Generally, the greatest potential of bias exists in the first row of Table 3.1, and the least potential of bias exists in the last row. Thus, the potential for organizational pressure is greater when the evaluator is employed by the organization whose program is being eval- uated than when the evaluator is employed by an outside agency. In addition, bias is more likely when the internal evaluator reports to the director of the program being evaluated than when the evaluator reports to someone outside that program. Sonnichsen (1999), the director of the internal evaluation unit at the FBI, argues that internal evaluators must be placed independently, separated from programs, to be effective. Lovell (1995), in commenting on internal evaluation, notes that, in the long term, the organization expects internal evaluation to pay off, that is, to provide recommendations for improved organizational operations. Bias that produces overly positive reports on programs leads to evaluation not fulfilling its promise.
Mathison (1999) has served as an internal and external evaluator and has written often on the issue. She believes that internal and external evaluators face the same ethical challenges but are part of different communities and that these
Chapter 3 • Political, Interpersonal, and Ethical Issues in Evaluation 103
communities influence their responses to ethical challenges. Internal evaluators, she asserts, operate within fewer communities than the external evaluator and their primary community is the organization in which they work. Consider, simply, the amount of time the typical internal evaluator spends in that organi- zation over weeks and years. External evaluators, in contrast, have many com- munities, which can include the organizations they evaluate, the organization that employs them, colleagues in evaluation and their professional association, funding agencies, and others. These communities, Mathison argues, influence evaluators’ ethical choices in many complex ways. For example, the internal evaluator’s closeness to the organization and relationships in it may enable her to behave more ethically when it comes to creating an ongoing evaluative culture in the organization or sustaining a dialogue about a controversial issue uncovered by an evaluation—a dialogue that may be required to bring about change. In con- trast, the external evaluator’s diversity of communities and greater distance from the community of the organization being evaluated makes it easier for her to raise questions concerning unethical issues in the program or in the organization. Mathison’s concept of the communities of reference for internal and external