What do evaluators using theory-based evaluation approaches do?
Bickman defines program theory as “the construction of a plausible and sensible model of how a program is supposed to work” (Bickman, 1987, p. 5). More recently, Donaldson defines program theory as “the process through which program components are presumed to affect outcomes and the conditions under which these processes are believed to operate” (2007, p. 22). In both cases, and in other definitions, program theory explains the logic of the program. How does it differ from a logic model? In fact, they are quite similar. A logic model may depict the program theory if its articulation of program inputs, activities, outputs, and outcomes is sufficient to describe why the program is intended to achieve its outcomes. Logic models are sometimes used as tools to develop program theory. In other words, a program theory may look like a logic model. In our experience, because the emphasis in logic models is on the stages of
162 Part II • Alternative Approaches to Program Evaluation
input, activities, outputs, and outcomes, the person developing the logic model focuses on listing all of the components in each of those categories and may, though certainly does not always, fail to describe the rationale or reasoning behind program success. In contrast, a program theory, while not containing prespecified categories such as inputs, activities, outputs, or outcomes, is intended to present the details of that reasoning. Bickman (1987) notes that program theory should clarify the relationship between the problem that the program assumes the student or client has and the program actions. Therefore, a program theory should begin with describing the client, or the assumptions the program has about the client, before the client begins the program. Logic models typically start a step after that, with program input. Another difference is simply that the organizations and people who write about logic models are different from those who write about program theory and theory-based evaluation.
Using Program Theory in Evaluation. The central element of theory-based evaluation approaches involves developing the theory for why the program should achieve its desired outcomes. It is this stage that distinguishes theory-based evaluations from other approaches. Chen (1990) differentiates between two models for developing program theory: (a) the stakeholder approach, in which the evaluator works with stakeholders, typically key program people, to discover their reasoning or underlying assumptions for program success; and (b) the social science approach, in which evaluators make use of their own knowledge both of the program and of the social science theory and research to develop a model.
Both Bickman and Chen note that developing program theory with stake- holders alone can often be problematic. Stakeholders may not know the program theory or, as Bickman writes, their theory may be “a vague notion or hunch” or “may be nothing more than a few simple assumptions about why the program should work” (1987, p. 6) because they are not well trained in social science theory or research. Or they may be purposefully obtuse or vague about program theory in order to obtain political support or funding or to avoid alienating an important group. Weiss, more recently, argues that improving the quality of program theory is one of the key challenges to evaluators. She notes that program managers or policymakers may or may not be able to develop a good-quality program theory. Program theories, she notes, must articulate the causal linkages between program actions and goals; if they do not, the theory is simply a model for implementation, a description of program processes. And, like Bickman, she expresses concern with relying too much on program stakeholders, planners, and practitioners to articu- late that theory. She observes, “Many of these theories are elementary, simplistic, partial, or even outright wrong” (Weiss, 1997, p. 78), and emphasizes the need for evaluators to combine stakeholder input with social science research to build sound program theory.
The process of developing program theory should, therefore, rely on a com- bination of input from stakeholders, theories and research from relevant social science studies, and the evaluators’ knowledge and expertise. Donaldson (2007), in his book on theory-based evaluation, decribes the steps for theory development
Chapter 6 • Program-Oriented Evaluation Approaches 163
more precisely than early writers, based on his experience in using theory-based evaluation in several large projects. His steps reflect this balance:
1. Engage relevant stakeholders. The evaluator talks with as many representatives as possible from different constituencies to get their views on the program, its intended long-term outcomes, and the process the program uses to achieve those outcomes.
2. Develop a first draft of program theory. This step is undertaken by the evaluator or the evaluation team.
3. Present the draft to stakeholders for further discussion, reaction, and input.
4. Conduct a plausibility check. Now, the evaluators consult existing research and evaluations that are relevant to the program theory to assess the plausibility of each link. Does research suggest that the link could occur as planned? That the program action could lead to the intended outcome?
5. Communicate these findings to key stakeholders and revise the program theory as needed. Donaldson observes that the plausibility check may suggest that serious program changes are needed or that stakeholders have been overly optimistic about the out- comes that might be achieved. Evaluators present research findings to stakeholders and work with them to revise the program theory and/or the program itself so that the model accurately represents what will be done and what can be achieved.
6. Probe arrows for model specificity. As in step 4, the evaluators take the lead in examining the program theory “at a deeper level of detail.” Donaldson indicates that at this stage the evaluators’ focus is typically on critical links and discussion of details such as the length of time required for the outcome to occur and the nature of the process. The purpose, here, is for the evaluation team to ensure that they have an accurate, in-depth understanding of the program as it is intended to proceed. Such understandings can influence, for example, when evaluators collect data on outcomes and the type of data they collect.
7. Finalize program impact theory. The stakeholders have the final say in approving the model that will serve as the foundation for studying the program. Donaldson notes that he prefers relatively parsimonious models, as do some stakeholders, but others prefer models with more detail (Donaldson, 2007, pp. 33–39).
This theory-development process precedes any decisions about the evaluation. In fact, Donaldson indicates that stakeholders often want to move into thinking about the evaluation and the implications of the program theory during the process of discussing the evaluation. However, it is important for the program the- ory to be fully developed before moving into identifying the evaluation questions to be answered or the methods to be used to address such questions. The program theory, its key principles, should not be influenced by how the evaluation will be conducted—for example, by worrying about how certain linkages would be tested—but should instead reflect a true picture of what the program is intended to do and how it will do it.
164 Part II • Alternative Approaches to Program Evaluation
The second phase of theory-driven evaluation is similar to that in most other evaluations. Working with stakeholders, the evaluators identify the key questions to be answered in the evaluation and the appropriate designs and methods for answering those questions. Although the theory-based evaluation approach has been considered a more quantitative approach because its proponents often come from quantitative areas, the approach does not prescribe or proscribe specific methods or designs. Often the emphasis is on testing the program model, that is, on questions of causality. The constructs identified in the program theory provide guidance as to what to measure and when to measure it (Lipsey, 1993). As Donaldson indicates, the selection of the evaluation questions to be addressed depend very much on the stage of the program (young versus mature enough to examine long-term outcomes) and what the stakeholders hope to learn. The development of program theory helps the evaluator to learn more about the program and its assumptions and, therefore, pro- vides the evaluator with critical information that can be used throughout the eval- uation. This includes determining what to study at different stages, identifying the constructs of interest, interpreting results, and making recommendations.
A typical model for theory-based evaluation would be to first study program implementation, focusing on whether key elements of the program theory are, in fact, delivered as planned. If so, the evaluator can then go on to study program outcomes knowing that this evaluation will be a test of program theory. If the program fails, it will mean that the theory does not work, at least with this client group in this context. But if the program is not implemented as planned, the evaluator may recommend changing the implementation to match the model, discarding the model as not feasible in this context, or trying some other model. In any case, the evaluation would not move on to measure outcomes because the program theory had not, in fact, been implemented. If outcomes were studied and success was achieved, the implementation study would demonstrate how the pro- gram that was delivered differed from the program theory. This modified program delivery might then become the standard model or program theory for the future.
Thus, the theory-based or theory-driven approach overcomes some of the failures of the objectives-oriented approach. It provides the evaluator with a way to look inside the black box and better understand what is happening between the time a student or client begins a program and when he or she concludes the program. With this information, theory-based evaluators argue, they can better test and determine the reasons for a program’s success or failure.