Other Applications of the Consumer-Oriented Approach
Product evaluation is also used by organizations and industries to evaluate prod- ucts at many different stages. Successful high-technology companies such as Apple have watched and studied consumers’ reactions to iPhones and Apple stores and used these data to make changes in their products, thus using consumer-oriented evaluations for formative purposes to revise their products. Amazon.com under- took a similar process with its electronic book, Kindle. Jonathan Morrell, an evaluator who has worked with industries to conduct many product evaluations, recently described the present-day use of product evaluations in industry. Although Scriven focused on product evaluations for summative, purchasing decisions by consumers, Morrell notes that most product evaluations in industries are formative in nature, as with the examples of Apple and Amazon.com. Evaluations take place through the product’s life cycle from initial design and the production process to marketing and circulation. The stakeholders for the evaluation include not only the managers of the organization and the consumers, but others associated with the product process as well. Morrell gives the example of pilots as a stakeholder for air- planes. Their opinions on human factors issues are important in creating a product that will permit them to perform optimally in flying the plane (Morell, 2005).
Influences of the Consumer-Oriented Approach: Uses, Strengths, and Limitations
As mentioned previously, the consumer-oriented approach to evaluation has been used extensively by government agencies and independent consumer advocates to make information available on hundreds of products. One of the best known
148 Part II • Alternative Approaches to Program Evaluation
examples in education today is the What Works Clearinghouse (WWC), begun in 2002 by the U.S. Department of Education’s Institute for Education Sciences (IES). (See http://ies.ed.gov/ncee/wwc.) WWC is a source for consumer-oriented evalu- ation information on the outcomes of educational programs and products. Its intent, like the consumer-oriented approach reviewed here, is to help consumers— teachers, school psychologists, and educational administrators—make choices about which educational products to use.
WWC differs dramatically, however, from Scriven’s more comprehensive evaluation process because its criteria for determining program success are confined to program outcomes, and its standards are concerned with research con- fidence in those outcomes. The stated mission of WWC is “to assess the strength of evidence regarding the effectiveness of the program.”6 Products studied using randomized control trials (RCTs) or regression discontinuity designs, which are viewed by IES as superior for establishing a causal link between the product or program and the outcome, receive the highest ratings. Studies using quasi-exper- imental designs may be endorsed with reservations. Scriven’s checklists and writ- ings argued for using several different criteria to reflect the elements of the product or program that were critical to successful performance. Although many of Scriven’s criteria concerned outcomes or performance (see his criteria for judging educational products listed previously), his process emphasized a comprehensive appraisal of the product, including need, side effects, process, support for users, and cost, as well as several criteria concerning outcomes or performance. WWC’s standards concern the extent to which the research establishes a causal effect, through preferred designs, between the program or product and the intended out- come. Although we bemoan the narrowing of the range of criteria and the stan- dards to assess those criteria, WWC’s efforts do prompt the potential user to consider the effectiveness of the program in achieving its outcomes and to provide a central location for accessing comparable information on educational programs and products. Educators are currently under much pressure to increase achieve- ment, and products can mislead in their marketing. However, WWC’s efforts to in- form the consumer about the demonstrated success of programs and products is today’s most successful application of the consumer-oriented approach in educa- tion in terms of visibility and number of users. Consumers can search the web site by area of interest, with topics including Early Childhood Education, Beginning Reading, Middle School Math, Dropout Prevention, and English Language Learn- ers. Many products are judged to have insufficient research evidence for a causal relationship between the product and the outcome. The only information provided on these products is the designation “no studies meeting eligibility
6In an ironic combination of consumer-oriented and expertise-oriented approaches, a blue-ribbon panel was convened in 2008 to determine whether WWC’s review process and reports were “scientifically valid” and “provide accurate information about the strength of evidence of meaningful effects in im- portant educational outcomes.” See http://ies.ed.gov/director/board/pdf/panelreport.pdf. Commenting that their charge was not to review the mission but to determine if the information was valid, the panel concluded that the information provided was valid.
Chapter 5 • First Approaches: Expertise and Consumer-Oriented Approaches 149
standards.” However, for products with studies meeting the eligibility standards, reports provide a brief description of the program or product, the research con- ducted on it, and a final judgment of its effectiveness at achieving the intended outcome.
Another prominent example of the consumer-oriented approach that illus- trates the overlap between it and the expertise-oriented approach are the test reviews of the Buros Institute of Mental Measurements. The Institute was founded in 1938 and has been conducting well-respected reviews of educational and psychological tests since that time. It currently produces two series: The Mental Measurements Yearbooks, now in its 17th edition, and Test Reviews Online (see www.unl.edu/buros). The Institute is consumer oriented in that it is “dedicated to monitoring the quality of commercially-published tests . . . promoting appropriate test selection, use, and practice” (http://www.unl.edu/buros/bimm/html/catalog. html, paragraph 1). It is designed to provide consumers with information on the quality of tests used in education and psychology. Each test review provides a brief description of the test and a discussion of its development and technical features, including reliability and validity information, a commentary, a summary, and ref- erences. However, the reviews contain elements of the expertise-oriented ap- proach because they are conducted by experts in psychometrics and, although the reviews make use of a prescribed format, the criteria and standards for reviewing each test and its competitors are not explicitly identified as would be done in Scriven’s approach. The Institute encourages its reviewers to use The Standards for Educational and Psychological Testing (1999), developed jointly by the American Educational Research Association (AERA), the American Psychological Associa- tion (APA), and the National Council on Measurement in Education (NCME), as a guide, but the Institute’s primary criterion for providing information on quality are in the selection of its expert reviewers.
Although the consumer-oriented evaluation approach continues to be used by magazines and web sites that review products, the approach is not one that continues to be discussed extensively in the professional evaluator litera- ture. However, Scriven’s writings on product evaluation in the 1970s, as well as Eisner’s writings on connoisseurship and criticism, were important in influenc- ing evaluation in its early stages to consider its role in valuing a program, policy, or product and in considering methods other than traditional social science research methods, for doing so. Each approach influenced evaluation practice today.
Major Concepts and Theories
1. The hallmark of the expertise-oriented evaluation approach is its direct reliance on professional judgment in the area of the program being evaluated.
2. Variations in the types of expertise-oriented evaluations include formal and informal review systems and ad hoc panels or individual reviews. These evaluations vary as to whether they are housed under an existing structure or organization, have
published standards that are used to evaluate the program or product, use a predeter- mined schedule for review, employ single or multiple experts, and directly affect the status of the program.
3. Accreditation systems in higher education, extending to K–12 schools, are a promi- nent example of the expertise-oriented evaluation approach in the United States and are currently in a process of discussion and change. Differences between the regional accrediting associations in the United States and the federal government concerning the purposes of these evaluations, the nature of the data collected or reviewed (outcomes, process, and inputs), the independence or neutrality of the expert evaluators, and the transparency of the process illustrate many of the controversies and political issues that can arise in expertise-oriented and other evaluations.
4. Elliot Eisner’s educational connoisseurship and criticism model made evaluators more aware of the skills of an expert, or connoisseur, in noticing critical dimensions of a product or program and in using methods outside of traditional social science measure- ment, especially qualitative methods of observation and description, to provide a complete picture of the program or product.
5. The consumer-oriented evaluation approach differs from the expertise-oriented approach in that it does not rely on content experts, or connoisseurs of the product, but rather on experts in evaluation. The approach is also based more centrally on evaluation logic and quantitative methods.
6. Michael Scriven, who wrote extensively about such evaluations, described the key steps as identifying the important criteria for judging the product or program, develop- ing standards to judge those criteria, collecting information or data, and synthesizing the information to make a final judgment that permits the consumer to compare the product with likely alternatives.
7. Both expertise-oriented and consumer-oriented approaches made evaluators aware of the importance of valuing in their work. It helped them recognize that the central task of evaluation is to make a judgment about the value of a program, product, or policy. The approaches advocate quite different methods for making that judgment and, therefore, each added separately to evaluators’ consideration of qual- itative methods and of criteria, standards, and checklists as potential methods for collecting data.
8. Both approaches continue to be used commonly by public, nonprofit, and private organizations and industries, but are not the subject of much writing in professional eval- uation today. The absence of evaluation literature on the subject is unfortunate. We hope evaluators will return their attention to these approaches commonly used by others to bring evaluative ways of thinking to the application of the approaches today.
150 Part II • Alternative Approaches to Program Evaluation
Discussion Questions
1. How do expertise-oriented and consumer-oriented evaluation approaches differ? How are they alike?
2. What do you see as the strengths of the expertise-oriented approaches? What are their drawbacks?
3. If a team of experts were reviewing your school or organization, what kinds of experts would you want on the team? What criteria would you want them to use to judge the quality of your organization?
4. Referring to question 3, who would you trust to make a better judgment—someone who is an expert in the content or subject matter of your organization or someone who knows evaluation theories, and methods for judging something? Justify your response.
5. Discuss the concept of a connoisseur. Are you a connoisseur at something? What is it? How does your experience with this thing help you to notice the important factors and be able to judge them better than a novice?
6. In consumer-oriented evaluation, what is the difference in criteria and standards?
7. How should one determine the criteria for evaluating a product? Should the focus be solely or primarily on outcomes? What should be the balance among the qual- ity of inputs (staff, facilities, budget), process (the conduct of the program), and outputs or outcomes?
Chapter 5 • First Approaches: Expertise and Consumer-Oriented Approaches 151
Application Exercises
1. What outside experts review your program or organization? a. If you work in an organization that is accredited, review the standards used for
accreditation. Do you feel the standards get at the real quality issues of the pro- gram or organization? What other standards might you add?
b. What are the areas of expertise of the evaluation team? Are they content experts, management experts, finance experts, evaluation experts, or experts in other ar- eas? How do you judge the mix of expertise? Might you add others? How might others judge their independence or objectivity in judging your organization?
c. If possible, interview those involved in the accreditation and learn more about the purposes of the accreditation (whether the emphasis is formative, summa- tive, or something else) and about how it has been used.
2. Your high school is going to be visited by an outside accreditation team. What issues do you think they should attend to? What do you think they might miss in a short visit? What information do you think they should collect? What should they do while they’re visiting? Do you think such a team could make a difference for your school? Why or why not?
3. Read a review of a restaurant, movie, or play that you have attended or seen. How does your opinion differ from the critic’s? How do the critic’s opinions influence your own? Does his or her expertise in the product (connoisseurship) or his ability to com- municate it (criticism) prompt you to think about the product in different ways?
4. Look at an evaluation of an educational product of interest to you on What Works Clearinghouse at http://ies.ed.gov/ncee/wwc. Critique their presentation of informa- tion from an expertise-oriented and from a consumer-oriented approach. What information is helpful? Would other information be helpful to you in making a decision? If so, what? Does that information relate to a different criterion or standard you have? How does the information fit into the approaches reviewed in this chapter?