Theory-Based or Theory-Driven Evaluation
Carol Weiss first discussed basing evaluation on a program’s theory in her 1972 clas- sic book building on earlier writings by Suchman (1967) on the reasons that pro- grams fail (Weiss, 1997; Worthen, 1996a). She has remained an effective and long-term advocate for theory-based evaluations (Weiss, 1995, 1997; Weiss & Mark, 2006). In the 1980s and 1990s, Huey Chen, Peter Rossi, and Leonard Bickman be- gan writing about theory-based approaches to evaluation (Bickman, 1987, 1990; Chen & Rossi, 1980; 1983; Chen, 1990). Stewart Donaldson (2007) is one of the prin- cipal evaluators practicing and writing about the theory-driven evaluation approach today.2 Edward Suchman (1967) had first made the point that programs can fail to
2Donaldson uses the term “theory-driven” but notes that the terms “theory-oriented,” “theory-based,” “theory-driven,” and even “program logic” and “logic modeling” are all closely related or sometimes interchangeable. We use the terms “theory-driven” and “theory-based” interchangeably, but attempt to use the words used by the author we are discussing.
Chapter 6 • Program-Oriented Evaluation Approaches 161
achieve their goals for two distinctly different reasons: (a) the program is not deliv- ered as planned and, therefore, is not really tested (implementation failure); and (b) the program is delivered as planned and the results, then, clearly indicate that the program theory is incorrect (theory failure). He and Weiss recognized that, if an eval- uation were examining whether a program achieved its goals and that program failed, it was important to know whether the failure was an implementation failure or a theory failure. With this information, the evaluator could then reach appropri- ate conclusions about the program and make useful recommendations for the deci- sion maker. To distinguish between implementation failure and theory failure, the evaluator had to know two things in addition to simply measuring outcomes: (a) the essentials of the program theory. and (b) how the program was implemented. With this information, the evaluator could then determine whether the program imple- mentation matched the theory. This was the beginning of program theory and the recognition of its importance to evaluation practice.
Chen’s and Bickman’s approaches to theory-based evaluation arose for these reasons, but also from their desire for evaluations to contribute more directly to so- cial science research knowledge. Chen, for example, argued that evaluators of the time erred in focusing solely on methodology and failing to consider the theory or tenets of the program. For many of those writing about theory-based evaluation as it first emerged in the late 1980s and 1990s, theory meant connecting evaluation to social science research theories. Chen (1990), for example, encouraged evaluators to search the scientific research literature to identify social science theories that were relevant to the program and to use those theories in planning the evaluation. Evaluation results could then contribute to social science knowledge and theory as well as to program decisions (Bickman, 1987). Thus, theory-based evaluation arose from a science-based perspective and was often considered to be a strictly quanti- tative approach by others during the debates on qualitative and quantitative meth- ods in the 1990s. However, today, theory-based evaluation is used by evaluators in many settings to gain a better understanding of the program. (See Rogers, 2000, 2001.) They can then use that understanding, the program theory, to better define the evaluation questions the study should address, to aid their choices of what con- cepts to measure and when to measure them, and to improve