Strengths and Limitations of Program-Oriented Evaluation Approaches
Probably the greatest strength and appeal of the objectives-oriented approach lies in its simplicity. It is easily understood, is easy to follow and implement, and produces information that program directors generally agree is relevant to their mission. This approach has caused program directors to reflect about their inten- tions and to clarify formerly ambiguous generalities about intended outcomes. Discussions of appropriate objectives with the community being served have given objectives-oriented evaluation the appeal of face validity—the program is, after all, merely being held accountable for what its designers said it was going to accomplish, and that is obviously legitimate.
Useful as this approach to evaluation seems to its many adherents, it has many drawbacks. Principal ones include the single-minded focus on objectives and their measurement. The focus on objectives can cause evaluators to ignore other important outcomes of the program, both beneficial and detrimental, and if the evaluation draws final conclusions, the judgment of the program may be seriously incomplete. The evaluator may focus on the objectives like a horse wearing blinders. He or she may ignore the road to the right that drops off precipitously or the absolutely breathtaking view to the left in its efforts to look at (and measure) only the goal to be reached—the objectives that have been ar- ticulated. The objectives-oriented approach also neglects program description, the need to gain an understanding of the context in which the program operates and the effects of that context on program success or failure. Finally, evaluators using this approach may neglect their role in considering the value of the objec- tives themselves. Are these objectives, in fact, important ones for the program and its clientele? The ethical principles of evaluation, in Guiding Principle E, re- quire the evaluator to consider “not only the immediate operations and outcomes of whatever is being evaluated but also its broad assumptions, implications, and potential side effects” (American Evaluation Association, 1995, p. 25). The objectives-oriented approach can appear seductively simple to novice evaluators who are only partially familiar with its philosophical and practical difficulties. Choices are involved in deciding which objectives to evaluate and how to interpret success or failure in each.
In today’s standards-based environment, evaluators have little authority over state-required tests to measure standards. However, evaluations concerning standards should help stakeholders consider which standards are appropriate for their students and what levels, when reached, will be considered a success. If test items do not fully reflect the objectives of a particular school or district, results from alternative measures can be provided as support for achievement to parents and community leaders. Such evaluations can open up discussions about standards and goals in education for different communities.
Evaluations that make use of logic models or program theory to learn more about the program and to shed light on what to evaluate and the appropriate means for doing so obviously overcome some of the criticisms to the objectives-oriented
Chapter 6 • Program-Oriented Evaluation Approaches 167
approach. Evaluators engage the stakeholders in dialogue so that they can learn more about the program, begin to develop a relationship with the stakeholders, and thereby gain a better understanding of what the evaluation might do. By the time a good theory-based evaluator moves to planning the evaluation, she should have a much better understanding of the values and concerns of the stakeholders regarding the program and the evaluation. The requirement for dialogue, as well as achieving an understanding and a clear articulation of the reasoning behind the program, is an obvious advantage of logic models and theory-based evaluations. These evaluators have focused on the program to identify and formulate the questions the evaluation will address, the timing of data collection, and the appropriate methods to be used. Some theory-based evaluators may feel obligated to evaluate the entire program theory, but that comprehensive focus is not the point of the process. The point is to understand the program from beginning to end and, then, to choose the appropriate links or components to evaluate given the stage of the program and the information needs of the stakeholders. Nevertheless, theory-based evaluators, like objectives-oriented evaluators, may struggle to see beyond their self-imposed blinders. Of course, theory-based evaluators are likely to focus on the theory and may ignore unintended program actions, links, outputs, or outcomes that merit attention. Further, their desire to test the theory as a whole may prompt them to neglect values or information needs of stakeholders. (See Fitzpatrick & Donaldson [2002] as Donaldson describes the pressures to evaluate the program theory for a training program for the unemployed that had been validated in Detroit, but now moved to a different population in California.)
Theory-based approaches are also criticized for oversimplifying the complexity of program delivery and context (Pawson, 2003). The reality of delivering programs is complex and, certainly, program theories simplify that complexity. But that is the purpose of theories or models—to reduce the messiness and complexity of actual program delivery to a parsimonious model that identifies the key assumptions or critical elements necessary for program success. In that way, the model helps the evaluator identify the most important elements or linkages to evaluate. Nevertheless, such reductionism does fail to convey the complexity of the real program and describing that complexity can also be an important role of evaluation. Oversimpli- fication often leads citizens and policymakers alike to fail to understand how diffi- cult, and costly, it is for programs or schools to achieve stated goals. Dahler-Larson calls for the need “to bridge the gap between conventions for relatively simple rep- resentations of causal models, on the one hand, and complex reality on the other” (2006, p. 152). He argues that theory-based evaluators should attempt to develop “different representations of program theories which will be fruitful in various ways depending on the purpose of the evaluation” (2006, p. 152).
We close our discussion of limitations of these types of evaluation approaches by briefly describing Scriven’s goal-free evaluation. His concerns with the limitations of objectives-oriented approaches led him to develop his now widely known proposals for goal-free evaluation (1972), still discussed today to make evaluators aware of the bias that a focus on particular program elements can impose. Although intentionally the opposite of objectives-oriented approaches, it seems logical to discuss this proposal here.