Classifications of Evaluation Theories or Approaches
Existing Categories and Critiques
In recent years, several evaluators have attempted to categorize evaluation theo- ries for different purposes. Shadish, Cook, and Leviton’s (1995) book was influ- ential in reviewing important evaluation theorists, at least partly to illustrate historical trends and changes in the field, but primarily to identify and describe important evaluation theories. Shadish et al. identified three stages of evaluation theory as it emerged in the United States. The first stage, in the 1960s, was char- acterized by a focus on using scientifically rigorous evaluation methods for social problem solving or studying the effectiveness of government programs at achiev- ing outcomes. The emphasis at this stage of evaluation was on examining causal effects of programs and, with this information, judging the value of each program. Shadish et al. focus on individual evaluators to illustrate the dominant theories at each stage. For the first stage, they profile Michael Scriven, who developed his the- ory of valuing—the process of reaching a judgment on the value of programs or policies—and Donald Campbell, who developed quasi-experimental methods to establish the causal effects of programs outside of the laboratory and discussed how these methods should be used by managers and evaluators. Stage two reflected evaluators’ growing concern with having evaluation results used.3
Evaluators’ focus on use in stage two prompted evaluation to grow and change in many ways, such as encouraging evaluators to establish relationships with specific stakeholders to facilitate use, and broadening the methods used to accommodate
2Kaplan (1964) described this fallacy by noting that, if you give a small boy a hammer, suddenly every- thing he encounters needs hammering. The same tendency is true, he asserts, for scientists who gain familiarity and comfort in using a particular method or technique: suddenly all problems will be wrested into a form so that they can be addressed in that fashion, whether or not it is appropriate. 3Stage one theorists had not written extensively about use, assuming results would naturally be used by consumers, managers, or policymakers.
Chapter 4 • Alternative Views of Evaluation 121
the potential information needs and values of different stakeholders. The theorists they profile in the second stage, Carol Weiss, Joseph Wholey, and Robert Stake, were concerned, in quite different ways, with increasing the responsiveness and utility of evaluations. In stage three, Shadish et al. view evaluators such as Lee Cronbach and Peter Rossi as integrating the first stage’s emphasis on truth or sci- entific validity with the second stage’s emphasis on evaluation’s utility to stake- holders. In efforts to have evaluation be both valid and useful, stage three evaluators introduce new concepts such as developing the theory of a social pro- gram to aid in its evaluation and adapt others.
Stufflebeam (2001b), too, analyzed evaluation theories or, what he, like us, calls “approaches.” His work was designed to reduce the burgeoning numbers of evaluation theories and to identify those with the greatest potential. He attempted to reduce the numbers of theories to those that are most useful by conducting an intensive study of 20 different evaluation approaches using some key descriptors to summarize each approach. He then used the Standards developed by the Joint Committee to judge nine approaches in more detail. His assessments of the vari- ous methods were also influenced by the extent to which each approach addresses what he sees as “evaluation’s fundamental requirement to assess a program’s merit or worth” (Stufflebeam, 2001b, p. 42). Of interest to us here is his catego- rization of the 20 approaches into three groups: (a) Question and/or Methods- Oriented approaches, (b) Improvement/Accountability approaches, and (c) Social Agenda/Advocacy approaches. His first category, Question and/or Methods- Oriented approaches, is the largest of the three groups, containing 13 of the 20 ap- proaches. These approaches, Stufflebeam notes, are alike in that they “tend to narrow an evaluation’s scope” by focusing on either particular questions or meth- ods (2001b, p. 16). Approaches in this category include ones that focus on partic- ular strategies to determine what should be evaluated (objectives-oriented and theory-based approaches), on particular methods to collect data (objective testing, performance testing, experimental studies, case studies, cost-benefit analysis) or to organize data (management information systems), or on a particular method for presenting and judging results (clarification hearing).4 Stufflebeam’s second category, Improvement/Accountability approaches, contains approaches that“stress the need to fully assess a program’s merit or worth” (2001b, p. 42). Stufflebeam sees these approaches as more comprehensive in their evaluation of programs in order to serve their purpose of judging merit or worth. Typical exam- ples include the accreditation/certification approach and Scriven’s consumer- oriented approach to judging the quality of products for potential consumers. The Social Agenda/Advocacy approaches, rather than having a primary emphasis on judging the overall quality of a product or relying upon a particular method, “are directed to making a difference in society through program evaluation” (Stufflebeam, 2001b, p. 62). In conducting evaluations, these approaches are concerned with involving or empowering groups who have less power in society. These
4These sub-categories are our own interpretation of the 13 approaches, not Stufflebeam’s.
122 Part II • Alternative Approaches to Program Evaluation
approaches include Stake’s client-centered or responsive evaluation and House’s deliberative democratic evaluation.
In 1985, Alkin and Ellett argued that to be considered a comprehensive the- ory, evaluation theories must address three issues: methodologies, how the data are valued or judged, and use of the evaluation. Later Alkin and House (1992) de- veloped these issues into three continua: (a) methods could be characterized along a continuum from qualitative to quantitative; (b) values could be characterized from unitary (one value or way of judging the data and program) to plural (many values); and (c) use could range from aspirations for instrumental, or direct use, to enlightenment or indirect use. In 2004, Alkin and Christie used these dimen- sions to categorize evaluation theorists and their approaches through the visual model of a tree. The roots of the tree reflect what they see as the dual foundations of evaluation: social inquiry (using a “systematic and justifiable set of methods”) and accountability and control (reflecting the purposes and intended use of eval- uation). The branches of the tree then reflect the three dimensions of methods, values, and use identified earlier by Alkin and House (1992). Individual theorists are placed on one of the three branches to reflect the key dimension of their approaches. Like Shadish et al. (1995), Alkin and Christie use individual evalua- tion theorists to illustrate different approaches to evaluation.
Each of these categorizations of evaluation approaches or theories provides useful insights into evaluation and its history and practice. Thus, Shadish, Cook, and Leviton illustrate the early focus on the truth that evaluation would bring to the judgments made about social programs, the later recognition that use needed to be consciously considered, and the integration and adaptation of the two issues in even later stages. Alkin and Christie’s model builds on these foundations iden- tified by Shadish et al. in slightly different ways. Its roots are in social inquiry, accountability, and control, but it considers evaluation’s emphases in three areas: methods, values, and use. Stufflebeam’s categories are different from the first two in that he focuses not on individual evaluators and their writings to identify cate- gories, but on the content of evaluation theories or models.5 He developed his cat- egories by considering the orienting devices or principles used for focusing the evaluation. The priorities used to focus the evaluation are reflected in his three categories: using particular evaluation questions or methods, taking a compre- hensive approach to making a judgment regarding quality of the program, or improving society and its programs by considering social equity and the needs of those with less power. Like Stufflebeam, our purpose is to reduce the current number of evaluation approaches. Although Stufflebeam’s method of reducing the approaches was to judge the quality of each, our synthesis of the approaches is intended to describe each approach to help you, the reader, to consider different
5Of course, examining the writings of proponents leads one to consider the theories as well because the individual is writing about his or her evaluation approach or theory. The difference is that Alkin and Christie (2004) and Shadish et al. (1995) were focusing on individuals to illustrate theories, and Stufflebeam’s writing is less concerned with individuals. Although some of the theories Stufflebeam reviews are identified with one individual, others are not.
Chapter 4 • Alternative Views of Evaluation 123
approaches and their potential use in your work.6 Although the many different approaches to evaluation can seem confusing, their diversity allows evaluators to pick and choose either the approach or the elements of an approach that work best for each program they are evaluating. Our task is to categorize the approaches in a way that helps you consider them and to expand your views of possible ways in which evaluations may be conducted.