The process of disciplined evaluation permeates all areas of thought and practice
Given its many uses, it may seem almost axiomatic to assert that evaluation is not only valuable but essential in any effective system or society. Citizens look to eval- uation for accountability. Policymakers and decision makers call on it and use it
Chapter 1 • Evaluation’s Basic Purpose, Uses, and Conceptual Distinctions 33
to make important decisions. Program staff can use evaluation to plan and improve programs to better meet clients’ and societal needs and to make decisions about how to stay within their budget. Consumers, such as parents, students, and voluntary clients, can make choices about schools for themselves or their children or the hospital, clinic, or agency they will contact for services. Evaluators can per- form many roles for those delivering programs. These include helping them develop good programs, helping them deliver the programs to changing clients for changing contexts, and helping them find interventions that are most successful in achieving their goals. Evaluators can help organizations as a whole through stimulating a learning culture, thereby helping those in the organization to ques- tion and consider their goals and their methods, their clients and their needs, and showing them how to use evaluative inquiry methods to meet their needs. As some evaluators note, evaluation plays an important continuing role in democ- racy. It informs citizens and, thus, empowers them to influence their schools, their government, and their nonprofit organizations. It can influence the power of stakeholders who have been absent from important decisions by giving them voice and power through evaluation. Scriven (1991b) said it well:
The process of disciplined evaluation permeates all areas of thought and practice. . . . It is found in scholarly book reviews, in engineering’s quality control procedures, in the Socratic dialogues, in serious social and moral criticism, in mathematics, and in the opinions handed down by appellate courts. . . . It is the process whose duty is the systematic and objective determination of merit, worth, or value. Without such a process, there is no way to distinguish the worthwhile from the worthless. (p. 4)
Scriven also argues the importance of evaluation in pragmatic terms (“bad products and services cost lives and health, destroy the quality of life, and waste the resources of those who cannot afford waste”), ethical terms (“evaluation is a key tool in the service of justice”), social and business terms (“evaluation directs effort where it is most needed, and endorses the ‘new and better way’ when it is better than the traditional way—and the traditional way where it’s better than the new high-tech way”), intellectual terms (“it refines the tools of thought”), and personal terms (“it provides the only basis for justifiable self-esteem”) (p. 43). Perhaps for these reasons, evaluation has increasingly been used as an instrument to pursue goals of organi- zations and agencies at local, regional, national, and international levels.
But, evaluation’s importance is not limited to the methods used, the stake- holder supplied with information, or the judgment of merit or worth that is made. Evaluation gives us a process to improve our ways of thinking and, therefore, our ways of developing, implementing, and changing programs and policies. Schwandt has argued that evaluators need to cultivate in themselves and others an intelligent belief in evaluation. He writes that “possessing (and acting on) an intelligent belief in evaluation is a special obligation of evaluators—those who claim to be well pre- pared in the science and art of making distinctions of worth” (2008, p. 139). He reminds us that evaluation is not simply the methods, or tools, that we use, but a way of thinking. Citing some problematic trends we see in society today, the political
34 Part I • Introduction to Evaluation
manipulation of science and the tendency to see or argue for all-or-nothing solutions that must be used in all settings in the same way, Schwandt calls for evaluators to help citizens and stakeholders use better means of reasoning. This better means of reasoning would draw on the kinds of thinking good evaluators should do. The characteristics of such reasoning include a tolerance for ambiguity, a recognition of multiple perspectives and a desire to learn from those different perspectives, a desire to experiment or to become what Don Campbell called an “experimenting society.” Describing this society and evaluation’s role in it, Schwandt writes:
This is a society in which we ask serious and important questions about what kind of society we should have and what directions we should take. This is a social environment indelibly marked by uncertainty, ambiguity, and interpretability. Evaluation in such an environment is a kind of social conscience; it involves serious questioning of social direction; and it is a risky undertaking in which we endeavor to find out not simply whether what we are doing is a good thing but also what we do not know about what we are doing. So we experiment—we see what we can learn from different ways of knowing. In evaluation, we try to work from the top down (so to speak) using what policy makers say they are trying to do as a guide, as well as from the bottom up, doing evaluation that is heavily participant oriented or user involved. All this unfolds in an atmosphere of questioning, of multiple visions of what it is good to do, of multiple interpretations of whether we as a society are doing the right thing. (2008, p. 143)
As others in evaluation have done, Schwandt is reminding us of what eval- uation should be. As evaluators, we learn how to use research methods from many disciplines to provide information and reach judgments about programs and policies, but our methods and theories underlie an approach to reasoning. This approach is its greatest promise.
Limitations of Evaluation. In addition to its potential for impact, evaluation has many limitations. Although the purpose of this book is to help the reader learn how to conduct good evaluations, we would be remiss if we did not discuss these limitations. The methods of evaluation are not perfect ones. No single study, even those using multiple methods, can provide a wholly accurate picture of the truth because truth is composed of multiple perspectives. Formal evaluation is more suc- cessful than informal evaluation, in part, because it is more cautious and more sys- tematic. Formal evaluation is guided by explicit questions and criteria. It considers multiple perspectives. Its methods allow one to follow the chain of reasoning, the evaluative argument, and to more carefully consider the accuracy, or the validity, of the results. But evaluations are constrained by realities, including some charac- teristics of the program and its context, the competencies of the evaluation staff, the budget, the timeframe, and the limits of what measures can tell us.
A more important limitation to evaluation than the methodological and fis- cal ones, however, are the political ones. We live in a democracy. That means that elected, and appointed, officials must attend to many issues. Results of evaluations
Chapter 1 • Evaluation’s Basic Purpose, Uses, and Conceptual Distinctions 35
are not their sole source of information by any means, nor should they be. Citi- zens’ input and expectations obviously play a role in decisions. Many stakeholder groups, experts, lawmakers, policymakers, and, yes, lobbyists, have information and experience that are important to consider. So, in the best of situations, evalu- ation is simply one piece of information, albeit an important piece, we hope, in the marble cake of sources used by decision makers in a democracy.
Finally, both evaluators and their clients may have been limited by a ten- dency to view evaluation as a series of discrete studies rather than a continuing system representing an approach to reasoning and personal and organizational growth. It can be difficult to question what you do and the activities that you be- lieve in, but evaluative inquiry must prompt us to do that, both in evaluating our evaluations (metaevaluation) and in evaluating programs. A few poorly planned, badly executed, or inappropriately ignored evaluations should not surprise us; such failings occur in every field of human endeavor. This book is intended to help evaluators, and the policymakers, managers, and all the other stakeholders who participate in and use evaluations, to improve their evaluative means of reason- ing and to improve the practice of evaluation.