A Classification Schema for Evaluation Approaches
We have chosen to classify the many different evaluation approaches into the four categories that we have developed based on our identification of the primary fac- tor that guides or directs the evaluation:
1. Approaches oriented to comprehensive judgments of the quality of the program or product: These approaches include expertise-oriented and consumer-oriented evaluations. They are the oldest approaches in evaluation, having been used by many before formal evaluation approaches were developed. We will dis- cuss Elliot Eisner’s writings on connoisseurship and criticism, accreditation, and other types of expertise-oriented evaluations and Michael Scriven’s consumer-oriented approach. The expertise and consumer-oriented approaches differ rather dramatically in who conducts the evaluation and the method- ology, but their commonality is that they direct evaluators to focus on valu- ing or judging the quality of the thing they are evaluating.
2. Approaches oriented to characteristics of the program: These approaches include objectives-based, standards-based, and theory-based evaluations. In each of these approaches, the evaluator uses characteristics of the program, its ob- jectives, the standards it is designed to achieve, or the theory on which the program is based to identify which evaluation questions will be the focus of the evaluation.
3. Approaches oriented to decisions to be made about the program: These approaches include Daniel Stufflebeam’s Context-Input-Process-Product (CIPP) ap- proach and Michael Patton’s Utilization-Focused Evaluation, as well as Joseph Wholey’s evaluability assessment and performance monitoring. These approaches focus on evaluation’s role in providing information to im- prove the quality of decisions made by stakeholders or organizations.
4. Approaches oriented to participation of stakeholders: These approaches include Robert Stake’s Responsive Evaluation, Practical Participatory Evaluation, Developmental Evaluation, Empowerment Evaluation, and democratically oriented approaches.
Placement of individual evaluation approaches within these categories is to some degree arbitrary. Several approaches are multifaceted and include characteristics that would allow them to be placed in more than one category. For clarity we have
6Although our purpose is not to judge the quality of each approach, but to introduce you to them, we do not include approaches that could not serve a valid purpose in an evaluation.
124 Part II • Alternative Approaches to Program Evaluation
decided to place such approaches in one category and only reference their other fea- tures in chapters where it is appropriate. Our classification is based on what we see as the driving force behind doing the evaluation: the factors that influence the choice of what to study and the ways in which the study is conducted. Within each category, the approaches vary by level of formality and structure, some being rel- atively well developed philosophically and procedurally, others less developed. Some are used frequently; others are used less, but have had a major influence on evaluators’ thinking. The following chapters will expand on these approaches.
Major Concepts and Theories
1. During evaluation’s relatively short history, many different approaches or theories concerning how to practice evaluation have emerged.
2. Evaluators should be familiar with the various approaches in order to choose the approach or elements of approaches most appropriate for the specific program they are evaluating and its context.
3. The different evaluation approaches were influenced by differing views of ontol- ogy (the world and reality) and epistemology (knowledge), and the methods for obtain- ing valid knowledge. These views often are associated with the evaluator’s graduate training and life experiences.
4. Today, prominent paradigms in evaluation and the social sciences include postpos- itivist, constructivist, transformative, and pragmatist paradigms.
5. Others have categorized evaluation theories or approaches according to a focus on truth and use and an integration of the two; by the categories of questions or methods, improvement/accountability, and social agenda/advocacy; and by their focus on meth- ods, values, or use.
6. We categorize theories based on the primary factor that guides the actions taken in the evaluation. Our categories include approaches that focus on making an overall judg- ment regarding the quality of the program, on program characteristics, on decisions to be made, and on stakeholder participation.
Discussion Questions
1. What are the key differences between the paradigms that have influenced evalua- tion? Which paradigm seems most appropriate to you? Why?
2. How can the ways in which one defines program evaluation impact an evaluation study?
3. What implications does the statement “evaluation is not a traditional discipline but a transdiscipline” have for the methodologies or approaches an evaluator may de- cide to use in an evaluation?
Chapter 4 • Alternative Views of Evaluation 125
Application Exercises
1. Think about how you could approach evaluation. Describe the steps you think you would follow. Then, analyze your approach according to your philosophical and methodological preferences. Explain how your background and what you would be evaluating could affect your approach. Describe other things that might affect your approach to evaluation.
2. Identify a program in your area that you would like to see evaluated. List some qualitative evaluation methods that could be used. Now list some quantitative methods that you see as appropriate. What might the different methods tell you?
3. The Anderson Public School District has recently begun a new training program for principals. What questions would you ask if you were to conduct an evaluation of this training program from the postpositivist paradigm? What types of data would you collect? How might this evaluation be conducted differently if you took a con- structivist perspective? A transformative perspective?