Nonevaluators Take on Internal Evaluation Responsibilities
Another change in evaluation in recent years concerns the number and types of people carrying out evaluation-related tasks. As evaluation expanded, many people— managers, supervisors, and other program professionals—began having responsi- bilities for evaluation as one part of their job. As noted in this history, evaluation has often been conducted by people without specific training in evaluation. Begin- ning in the 1960s when social science researchers began conducting evaluation studies to meet the demand, evaluation has often had to rely on those without specific education or training in evaluation to conduct studies. In earlier years, those people were often social scientists who had training in methodology and research, but were not familiar with evaluation theories and particular concerns about context and use. Social science researchers continue to conduct evaluations today. However, many learn about the discipline of evaluation and supplement their methodological expertise with further reading, training, and attendance at evalua- tion conferences, as the discipline of evaluation grows and becomes better known. New today are the increasing numbers of managers and program staff who lack the methodological training in evaluation and in social science research methods, but are often responsible for internal evaluations (Datta, 2006).
Evaluation in the nonprofit sector provides an excellent example of the extent to which in-house evaluators, typically program managers and staff with other program responsibilities, have also become responsible for major compo- nents of data collection and evaluation in their organizations. More than 900,000 nonprofit and religious organizations deliver the majority of social service pro- grams in the United States (Carman, Fredericks, and Introcaso, 2008). Most of these organizations receive funds from the 1,300 local United Way organizations, and United Way requires these organizations to conduct evaluations of their programs. The United Way approach to evaluation has admirable components, including sig- nificant training, but most of the evaluations, with United Way encouragement, are conducted by existing staff with occasional guidance from external evaluators (Hendricks, Plantz, and Pritchard, 2008). Hendricks et al. (2008), who are otherwise pleased with many elements of the United Way approach, are concerned that the overreliance on current employees who lack evaluation expertise may short- change the organizations when it comes to effective use of the results. Survey
52 Part I • Introduction to Evaluation
studies of evaluators provide further evidence of the increase in numbers of eval- uators who are both internal to the organization and have other responsibilities within the organization. Christie (2003) found that many of the evaluators she surveyed in California were internal and held other, generally management, responsibilities. Many had little or no training in evaluation and were unfamiliar with evaluation theories and approaches.
In education, school districts have been faced with serious budget constraints and many have coped with these fiscal constraints by cutting central office staff, including evaluation departments. Schools, faced with increasing evaluation demands in the current standards-based environment, have had to cope with these demands with fewer evaluation professionals. As a result, teachers and adminis- trators often face additional evaluation responsibilities. The expansion of evaluation has, therefore, had some unintended consequences that have implications for building organizational capacity and for improving education and training.
Many people involved in conducting in-house evaluations have primary professional identifications other than evaluation. They are often not interested in becoming full-time evaluators and, hence, university-based education is not the best option for providing training for these individuals. (See Datta [2006] for her discussion of the need to learn more about the evaluations produced by these practitioners to consider how their training needs can be addressed.) Expanded training opportunities and creative thinking by those in the evaluation field are needed to help these people develop their evaluation skills. Evaluation kits abound, but often focus on basic methodological issues such as designing a survey, and not on critical issues such as carefully defining purpose, involving stakeholders, and considering use.
Although the explosion of employees in organizations conducting evaluation has serious implications for training and for the accuracy, credibility, and use of eval- uation studies, the move to involve other employees of schools and organizations in evaluation also has great advantages. In 2000 and 2001, the conference themes of both presidents of the American Evaluation Association addressed, in different ways, the issue of working with other employees in organizations to improve eval- uation quality and use. Noting the increasing demand for evaluation and, yet, evaluators’ continued struggles in affecting programs and policies, Laura Leviton, the president of AEA in 2000, used her theme of “Evaluation Capacity Building” to discuss ways to build evaluators’ collective capacity to conduct better evaluations. Her suggestions included recognizing and using the strengths of program practi- tioners in program logic and implementation, in organizational behavior, and in the people skills needed to help those within organizations understand and use evaluation (Leviton, 2001). Rather than maintain a distance from managers and others in the program being evaluated as some evaluators have done, Leviton encouraged eval- uators to learn from these people with experience in the organization. James Sanders, our co-author and AEA president in 2001, chose as his theme, “Main- streaming Evaluation.” In his opening remarks, Sanders noted that when he and Blaine Worthen published the first edition of this book in 1973, they began with the observation that “evaluation is one of the most widely discussed but little used
Chapter 2 • Origins and Current Trends in Modern Program Evaluation 53
processes in today’s systems” (2002, p. 253). He notes that the status of evaluation has improved but that it is still not second nature to organizations. Explaining his concept of mainstreaming evaluation, Sanders said, “Mainstreaming refers to the process of making evaluation an integral part of an organization’s everyday operations. Instead of being put aside in the margins of work, evaluation becomes a routine part of the organization’s work ethic if it is mainstreamed. It is part of the culture and job responsibilities at all levels of the organization” (2002, p. 254). Today, with much attention being paid to evaluation and accountability and with many managers and other employees playing a part in conducting evaluations, we have that opportu- nity. As noted earlier, the spread of evaluation responsibilities has its risks, but it also has potential benefits to evaluation and to the organization. We can cope with the risks by expanding training opportunities and by making use of partnerships between internal and external evaluators, as discussed in Chapter 1. Meanwhile, the fact that many employees of organizations, schools, and other agencies who do not identify themselves as evaluators are now involved in evaluation presents an opportunity for evaluation to become part of the culture of the organization. But, this will succeed only if we proceed carefully. Just as social scientists who came to evaluation in the 1960s often erred in viewing evaluation as simply the application of research methods in the field, today’s busy managers or professionals who are conducting evaluation while balancing other responsibilities in their organization may view evaluation as simply collecting some data and reporting it to others. Sanders’ concept of mainstreaming evaluation includes carefully crafting the pur- poses of evaluation for organizational learning and use.