The following are major characteristics of goal-free evaluation:
• The evaluator purposefully avoids becoming aware of the program goals. • Predetermined goals are not permitted to narrow the focus of the evaluation
study. • Goal-free evaluation focuses on actual outcomes rather than intended
program outcomes. • The goal-free evaluator has minimal contact with the program manager
and staff. • Goal-free evaluation increases the likelihood that unanticipated side effects
will be noted.
It might be helpful to point out that objectives-oriented and goal-free evaluations are not mutually exclusive. Indeed, they supplement one another. The internal staff evaluator, of necessity, conducts a goal-directed evaluation. She can hardly hope to avoid knowing the goals of the program, and it would be unwise to ignore them even if she could. Program managers obviously need to know how well the program is meeting its goals, and the internal evaluator uses goal-directed evalu- ation to provide administrators with that information. At the same time, it is important to know how others judge the program, not only on the basis of how well it does what it is supposed to do, but also on the basis of what it does in all
Chapter 6 • Program-Oriented Evaluation Approaches 169
areas on all its outcomes, intended or not. This is a task for the external goal-free evaluator who knows nothing of the program goals. Thus, goal-directed evalua- tion and goal-free evaluation can work well together. And, while the major share of a program’s evaluation resources should not go to goal-free evaluation, it is unfortunate when all resources go to goal-directed evaluation on a program when the stated goals do not even begin to include all of the important outcomes.
Major Concepts and Theories
1. The objectives-oriented evaluation approach was one of the first approaches to evaluation and is still commonly used today. Some of its present-day forms are standards- based testing and accountability in education and performance monitoring systems used in many government programs.
2. The objectives-oriented approach focuses on articulating the objectives of a program and collecting data to determine the extent to which they are achieved. Ralph Tyler and Malcolm Provus were early advocates of different facets of objectives-oriented evaluation.
3. Logic models are often used by program managers and evaluators today to link program inputs, processes, outputs, and outcomes and can serve as the foundation for making decisions about program or evaluation activities.
4. Theory-based or theory-driven evaluation approaches make use of social science theories and research relevant to the program and stakeholders’ assumptions about why the program should work, to develop program theories. These program theories then serve as a foundation for selecting evaluation questions and making decisions about what to study and when to collect data on it. Theory-based evaluations are a frequently used approach in evaluation today.
5. The goal-free evaluation approach was proposed primarily to identify any unan- ticipated side effects of a program that a goal-directed or objectives-oriented evaluation might miss because of the focus on the intended program outcomes rather than on the actual program outcomes.
Discussion Questions
1. What are some of the primary reasons for using an objectives-oriented evaluation approach? Name a program or policy that you think would be useful to evaluate with this approach and discuss your rationale.
2. How does Provus’s discrepancy model add to your thinking about what evaluation might do? Can you think of a program or policy that might be usefully evaluated with this approach?
3. What do you see as the advantages and disadvantages to the theory-based approach?
4. You only have a finite amount of money to spend on evaluation in your organization, but you’re free to use it in any way you would like. Would you rather use it for an
170 Part II • Alternative Approaches to Program Evaluation
on going performance monitoring system or for two or three in-depth evaluation studies each year that focus on particular programs or problems?
5. All program-oriented approaches share the strength of focusing the evaluator on some key characteristics of a program. They also share the weakness of perhaps causing the evaluator to focus unduly on those elements and, hence, to ignore other program effects, desirable or undesirable. Do you think the strength is worth the drawbacks? That is, would you rather focus on specific program characteristics (objectives or elements identified in a logic model or program theory) or would you rather try a form of goal-free evaluation, evaluating the program with no knowledge of its intended purposes? Justify your choice.
Application Exercises
1. Cheryl Brown is a program administrator for a state department of social services. She has been responsible for implementing a parenting program in which the goal is to reduce the incidence of child abuse and child neglect. To evaluate the program, she decides that she will depend on one performance measure—the number of cases of reported abuse and neglect. Use what you have just learned about Tyler’s approach to evaluation, Provus’s Discrepancy Evaluation Model, logic models, and goal-free evaluation to expand the evaluation design for this program. What are the risks of basing the evaluation on just this one measure? What are the advantages?
2. Jane Jackson is a leader of the English faculty of Greenlawn Middle School, which is under considerable pressure to reduce the achievement gap in writing that exists between Caucasian and minority students. It is May and she and the other English teachers are meeting to consider what data might help them examine their writing program in the next year and reduce the achievement gap in writing. Using what you have learned about different approaches in this chapter, advise Ms. Jackson on their evaluation. What elements of the approaches you have learned about might they use? How could she organize her evaluation? In particular, how might discussion of objectives, discrepancies, cubes, logic models, or program theory help them?
3. Many school districts and government agencies today are considering some form of performance-based pay for their employees. You are preparing to meet with a school district to discuss an evaluation of its performance-based pay program. You are considering using a theory-based approach to learn more about the key assumptions of this program. What do you think would be the advantages and disadvantages of using this approach to evaluate the program? With your classmates, develop a possible theory for such a program.