Here are a few of our suggestions for establishing and maintaining good com- munications during an evaluation:
1. In planning the evaluation—writing the proposal or preparing the contract—build in time for communication. Remember to include time for communication through meet- ings, meetings, and more meetings! Discuss evaluation plans and results orally with key groups first. Allow for dialogue. Listen to how the different individuals you are meeting with react to the evaluation plans and, later, to the results. Communication with others, of course, should not always be in a group setting. The evaluator should remember to take time to chat with those delivering or managing the program when on site. Learn what their concerns are about the evaluation and about the program itself. What are they worrying about? What pressures are they under? Use interim reports and memos to send information to individuals or groups whom you may not encounter frequently, but follow up by phone or in person to talk with them and get their thoughts. The evaluator needn’t be co-opted by seeking to communicate with these groups and to hear their ideas. But hearing their ideas, their perspectives, and their experiences with the program and the evaluation is the only way that the evaluator can break down barriers to evaluation and prepare stakeholders to receive the results and see them as credible and useful.
2. Prepare clients (those who sponsor the evaluation) and other stakeholders for evalua- tion. Develop an “evaluation spirit” by talking with all participants about the purpose and benefits of the evaluation. Resistance to evaluation comes naturally to most
Chapter 3 • Political, Interpersonal, and Ethical Issues in Evaluation 77
people and not knowing what to expect can only increase such resistance. If stakeholders are new to evaluation, or have had previous bad experiences with it, learn more about their concerns and fears. Ask about their previous experiences with evaluation and what they think this one will do. Let them know your views about evaluation and what it can do. As appropriate, provide stakeholders with in- formation on other evaluations or evaluation approaches or on organizational change and learning to illustrate what evaluation and self-examination can accomplish for an organization. Current literature on continuous improvement and learning organizations can be helpful. (See Chapter 9 for a discussion of using evaluation to improve learning in organizations and how to use the research on learning to improve evaluation practice.)
3. Invite and nurture outside participation. In evaluating a school program, for example, remember that parents, school board members, and citizens in the community are all potential stakeholders. Their participation not only strengthens the evaluation but also signals that this is an important project. When evaluating programs in health and human services or in corporate settings, the external stakeholders may be different (e.g., citizen groups, families of those receiving treatment, county commissioners, service providers, corporate board members, consumer advocate groups). Learn who the stakeholders are for the program. Eval- uators can play an important role in seeking to empower previously disenfran- chised groups by bringing representatives of these groups into the discussion. (See our discussion of participatory and empowerment approaches in Chapter 8.)
4. Seek input from key individuals or groups on evaluation decisions. Evaluators should not make important decisions alone. While evaluators may be the most expert in methods of data collection, the client and stakeholders have expertise in the program being evaluated and their experiences with it. Their needs and views must be sought and considered. Foster a spirit of teamwork, negotiation, and compro- mise. Seek input and consult with others at important points, including determin- ing the purpose of the evaluation and developing the evaluation questions, selecting sources and methods for collecting data, developing measures or looking for existing measures, analyzing and interpreting the data, and, of course, consid- ering the implications of the findings. The evaluator should frequently seek input from others on when to disseminate results (not waiting until all is done) and how to do so. Others will know what is most likely to be heard or read, when individu- als or groups are interested in results, and which results would be of most interest to them. Watch out for political agendas, though. Don’t make assumptions about what people want to know. Instead, talk with them to find out.
5. Encourage constructive criticism of the evaluation. Invite stakeholders to challenge assumptions or weaknesses; encourage divergent perspectives. Model a spirit of fair- ness and openness when critical feedback is given. By encouraging stakeholders to provide constructive, critical feedback on their work and then responding in an accepting and open manner, evaluators can demonstrate the evaluation spirit they hope to see in stakeholders. (See Fitzpatrick and Donaldson [2002] for a discussion
78 Part I • Introduction to Evaluation
of Donaldson’s use of 360-degree feedback during an evaluation to provide a means for program people to comment on the evaluation, just as the evaluator is com- menting on the program.)
Following these recommendations can improve responsiveness to the evaluation— and, hence, its subsequent use—and can enhance the quality of the evaluation prod- uct itself.
Maintaining Ethical Standards: Considerations, Issues, and Responsibilities for Evaluators
Given that evaluation occurs in a real-world political context and that to carry out a good evaluation, evaluators must learn about the political context and develop good communications with stakeholders, it should not be surprising that ethical problems can often arise for the evaluator. In our discussion of the political context, we noted the political pressures that can emerge to change the purpose of the eval- uation or an evaluation question, to select data sources or designs that are more likely to produce desired results, and, of course, to interpret or report findings in more favorable or desired ways. In addition, as one works to improve communica- tion with clients and stakeholders, closer relationships develop and these relation- ships can present ethical problems. Therefore, evaluators must be sufficiently sensitive to the potential ethical problems that can occur in evaluation so that they recognize the problems when they occur and have some sense for what to do about them. One step in that direction is gaining knowledge about the profession’s expectations for ethical behavior in evaluation.
Let us begin this important section on ethical behavior in evaluation with a real-world example of ethical failures in the evaluation of a different type of prod- uct. In 2009, the world was facing what some were calling a “financial meltdown.” Home values and the stock market were plummeting. Thousands had to leave their homes because of foreclosures. Unemployment was increasing and predicted to reach 10% in the United States. Countries all over the world were affected by the crisis. Though the factors that contributed to the financial meltdown were many, analysts and elected officials were highly critical of the role of rating agencies in this crisis. Rating agencies such as Moody’s Investors Service, Fitch, and Standard & Poor’s analyze stocks and bonds and assign credit ratings based on their research. Dating back to the early years of the twentieth century, these agencies began con- ducting research to judge the quality of companies that issue bonds and, through their ratings, to provide information to investors who made decisions based on these ratings, that is, deciding that a company is safe or unsafe for investment. But changes occurred in recent years that affected the quality of these ratings. For more than 50 years, investors paid these companies for their ratings. As the economy worsened in the 1970s, companies issuing bonds began paying agencies for their own ratings. This established a huge, but relatively unnoticed, conflict of interest.
Chapter 3 • Political, Interpersonal, and Ethical Issues in Evaluation 79
Rating agencies were now rating the bonds of those who paid them. Of course, some bonds continued to receive low ratings, but many were highly rated. In 2007, 37,000 “structured finance products,” the complex financial products that are the source of much of the financial turmoil today, received the highest possible ratings (“A Brief History of Rating Agencies,” 2009). Today, many of those ratings have been downgraded, but too late for many investors and citizens who are paying for company bailouts. The first suggestion of problems with the rating systems arose in 2001 when Enron Corporation, which had earlier been highly rated by these agencies, defaulted. In 2007, the heads of these rating agencies were called to testify before the U.S. Congress and were heavily criticized for their failure to identify risky investments and warn investors.
Evaluators, like analysts at Moody’s, Standard & Poor’s, and Fitch, judge the quality of something and provide information for clients and other stakeholders to make decisions based on those judgments. Although our actions are unlikely to collectively threaten the financial stability of the world economy, many elements of our work are very similar. Clients and others—the public—look to evaluators to provide objective, independent judgments about the quality of programs, prod- ucts, or policies. We use analytic methods to judge the product and assume that the transparency and validity of those methods will substantiate our findings. But just as employees at the rating agencies must interact with real people at the com- panies they are judging to conduct their analyses, we, too, interact with clients and stakeholders to learn about the programs or policies we are judging. In many cases, our client is the CEO or a manager of the program we are evaluating. Our results may be used by our client or the manager of the program to seek further funding or to make decisions about funding a program, just as the results of bond raters are used by investors and company managers to make decisions about fund- ing a company. The potential for ethical conflicts—conflicts we may not see— is great. The conflicts lie not simply in methodological choices, but in the relationships that develop when research methods are used in the real world. In this section, we will describe some of the ethical problems that evaluators encounter, discuss ethical codes developed to guide practice, and provide some suggestions of our own to help evaluators consider how to behave ethically.