Strategies for Reducing Bias.
What can an evaluator do to minimize the bias that personal views and experience bring to an evaluation? One strategy recom- mended by qualitative researchers (Lincoln and Guba, 1985, Miles & Huberman, 1994; Schwandt & Halpern, 1989) is to maintain an “audit trail,” which Schwandt defines as “a systematically maintained documentation system” (2001b, p. 9) to record all the details of the process of conducting the study. The audit trail would include the evaluator’s notes on evolving perceptions, day-to-day procedures, methodological decisions, day-to-day personal introspections, developing in- sights and hypotheses to help the evaluator explore how the evaluation design is emerging and the values and experiences that may influence the evaluator in that evolution. (See Cooksy [2000] for an excellent example of using such memos to aid in reflecting on an ethical problem encountered in data collection.) The eval- uator may choose to use the notes for self-reflection and consideration of how
Chapter 3 • Political, Interpersonal, and Ethical Issues in Evaluation 99
values and experiences may be introducing bias. Alternatively, the evaluator may decide to share portions of the notes with an external party. This person, generally another evaluator, can review the audit trail to explore the appro- priateness of the evaluation decisions and the ways in which bias may have been introduced.
Another strategy for minimizing bias is through the process of metaevaluation, or the evaluation of an evaluation in which an outside person reviews an evaluation for its quality. This topic is addressed in detail in Chapter 16. Whatever methods are used, it is important for evaluators (and clients) to examine their personal values and beliefs and consider how these factors can influence their approach to each evalua- tion and to their eventual conclusions and judgments. Becoming aware represents the first step in preventing bias.
Interpersonal Relationships and Bias. It is apparent to even the casual observer that individuals’ feelings toward one another can color their judgments, not only about each other but about practically anything with which the other person is perceived to be associated. Hence, we have legal restrictions on testimony about one’s spouse and anti-nepotism policies that prohibit individuals from being in positions where they would need to make decisions about the salary, promotion, or job security of a family member. Similarly, evaluators should avoid evaluating programs that a close friend or family member is concerned with, whether as a policymaker, a manager, or a person delivering the program. The apparent conflict of interest would be too strong even if the evaluator were able to overcome the bias the interpersonal relationship introduced.
Internal evaluators, except in the largest organizations, are almost inevitably evaluating programs that are staffed by someone they know. Therefore, internal evaluators need to think carefully about how to define their role in such settings. Even if the purpose of the evaluation is formative or for organizational learning, the evaluator needs to be prepared to give negative feedback. To achieve change, that feedback may be given in a way that is clear but palatable. Nevertheless, evaluators should be alert to examining how their relationships with those who operate or manage the program can influence the choices and decisions made. Such relation- ships can affect many elements of the evaluation, from the questions the evaluation addresses to the ways in which results are interpreted and presented. As an evalua- tor, you are hired or assigned to provide an independent, impartial judgment, and concerns about personal relationships should not interfere with the evaluator’s responsibility.
As we discussed earlier in this chapter, however, evaluators have a respon- sibility to develop some type of relationship with the client and stakeholders con- cerned with the evaluation. They must be able to communicate with them effectively so they can understand their needs and provide information in a way that meets those needs. Evaluators who are entirely new to the setting of the evaluation should spend time observing the program, meeting with clients and stakeholders, and developing relationships. These relationships are intended to help the evaluation to succeed—to reduce mistrust, to improve understanding, and so
100 Part I • Introduction to Evaluation
forth—but these relationships also introduce bias. Evaluators are likely to feel more comfortable with people whose values and beliefs are like their own, who support the evaluation and who are open to its methods and interested in its re- sults. At the same time, evaluators learn that some people concerned with the evaluation are more difficult. They are suspicious, accusatory, demanding, inflexible, or behave in any number of ways that are frustrating to the evaluator. These relationships, good and bad, influence the evaluator’s behavior. Is the eval- uator prepared to give tough, negative results—ones the evaluator knows they won’t like—to people with whom he has established rapport? To those who are helpful to the continuation of the study? These are tough issues, but evaluators must be prepared to deal with them, to prepare audiences for difficult results, and to prepare themselves for delivering them.
We find it is useful to clarify and demonstrate one’s role at the beginning of the evaluation, during the planning phase. Evaluators do not need to be the tough guy but they do need to be willing to ask tough questions and provide difficult feedback. In a later chapter, we will discuss using logic models during the early stages of the evaluation, as a method to help the evaluator understand the program. This is often a useful time to ask probing or tough questions such as, “Now, why is it that you think that this activity will lead to X change? Some of the research I have read doesn’t support that,” or “Which of your objectives do you think you are probably not achieving?” You may choose other ques- tions, but our point is that at the beginning of the study—not waiting until the end—you should start to define your role as someone who is interested in them and their program, but is also curious, objective, and questioning. This persona or manner then becomes part of your interpersonal relationship with others in the program.