How Program-Oriented Evaluation Approaches Have Been Used
The objectives-oriented approach has dominated the thinking and development of evaluation since the 1930s, both in the United States and elsewhere (Madaus & Stufflebeam, 1989). Its straightforward procedure of using objectives to deter- mine a program’s success or failure and to serve as a foundation for program
Chapter 6 • Program-Oriented Evaluation Approaches 165
improvements, maintenance, or termination of program activities has proved an attractive prototype.
In education, the approach influenced the development of taxonomies of educational objectives (Bloom, Hastings, & Masia, 1971), the criterion-referenced testing movement of the 1960s and 1970s, and today’s standards-based movement. As we noted in Chapter 2 in describing current trends in evaluation, the focus of evaluation today is on measuring outcomes; in schools, that takes the form of educational standards. No Child Left Behind (NCLB), the legislation passed by Congress in 2001, required all states that had not already done so to develop rigorous content standards for learning and tests to be given annually to measure accomplishment of those standards. This objectives-oriented means of evaluation now dominates K–12 education. Annual measurable objectives (AMOs) are used as a means of measuring progress toward the standards.
The objectives-oriented tradition has also influenced evaluation and management practices from the 1960s when Robert McNamara and the Rand Corporation brought Planning, Programming, and Budgeting Systems (PPBS) to the U.S. Defense Department, Management by Objectives (MBO), outcome monitoring (Affholter, 1994), and the Government Performance and Results Act (GPRA) (National Performance Review, 1993). Today, the Program Assessment Rating Tool (PART) has replaced GPRA (Office of Management and Budget, 2004), and performance monitoring has become a mainstay of many government man- agement systems. Dahler-Larson (2006), in commenting on trends in evaluation today, sees performance monitoring as today’s version of objectives-oriented or goal-oriented evaluation approaches. He observes that in-depth, goal-oriented evaluations have been replaced by monitoring of performance indicators. Perfor- mance monitoring systems are used by managers to monitor progress toward results. The systems may look at outputs, productivity, efficiency, service quality, customer satisfaction, or outcomes, but the focus is on the program and on the results (Positer, 2004). Logic models are sometimes used to identify the critical elements in the system that should be monitored, but to make it feasible to monitor these elements in an ongoing way requires compromises. So, data for performance monitoring systems tend to be solely quantitative and cost-effective to collect. Rarely are true long-term outcomes measured in an ongoing performance monitoring system if those outcomes are at all complex.
Although versions of objectives-oriented approaches continue to be popu- lar with many government agencies and foundations for intensive evaluations, theory-based evaluation approaches are often the approach of choice by professional evaluators, particularly those with a more scientific bent. Many government funding agencies, particularly at the federal level in the United States, require programs to articulate their program theory or logic model. In addition, foundations such as the Aspen Institute, with their work on compre- hensive community initiatives designed to have an impact at the community level, have pursued theory-based evaluations as a way to help them articulate the theory of complex programs and, then, to evaluate that theory as imple- mented (Weiss, 1995).