Internal and External Evaluations
The adjectives “internal” and “external” distinguish between evaluations conducted by program employees and those conducted by outsiders. An experimental year- round education program in the San Francisco public schools might be evaluated by a member of the school district staff (internal) or by a site-visit team appointed by the California State Board of Education (external). A large health care organization with facilities in six communities might have a member of each facility’s staff evaluate the
F oc
u s
of Q
u es
ti on
s
28 Part I • Introduction to Evaluation
TABLE 1.4 Advantages of Internal and External Evaluators
Internal External
More familiar with organization & program history
Can bring greater credibility, perceived objectivity
Knows decision-making style of organization
Typically brings more breadth and depth of technical expertise for a particular evaluation
Is present to remind others of results now and in future
Has knowledge of how other similar organizations and programs work
Can communicate technical results more frequently and clearly
effectiveness of their outreach program in improving immunization rates for infants and children (internal), or the organization may hire a consulting firm or university research group to look at all six programs (external).
Seems pretty simple, right? Often it is, but how internal is the evaluation of the year-round school program if it is conducted by an evaluation unit at the cen- tral office, which is quite removed from the charter school implementing the pro- gram? Is that an internal or external evaluation? Actually, the correct answer is both, for such an evaluation is clearly external from the perspective of those in the charter school, yet might be considered an internal evaluation from the perspec- tive of the state board of education or parents in the district.
There are obvious advantages and disadvantages connected with both internal and external evaluation roles. Table 1.4 summarizes some of these. Internal evalu- ators are likely to know more about the program, its history, its staff, its clients, and its struggles than any outsider. They also know more about the organization and its culture and styles of decision making. They are familiar with the kinds of informa- tion and arguments that are persuasive, and know who is likely to take action and who is likely to be persuasive to others. These very advantages, however, are also disadvantages. They may be so close to the program that they cannot see it clearly. (Note, though, that each evaluator, internal and external, will bring his or her own history and biases to the evaluation, but the internal evaluators’ closeness may pre- vent them from seeing solutions or changes that those newer to the situation might see more readily.) While successful internal evaluators may overcome the hurdle of perspective, it can be much more difficult for them to overcome the barrier of posi- tion. If internal evaluators are not provided with sufficient decision-making power, autonomy, and protection, their evaluation will be hindered.
The strengths of external evaluators lie in their distance from the program and, if the right evaluators are hired, their expertise. External evaluators are perceived as more credible by the public and, often, by policymakers. In fact, external evaluators typically do have greater administrative and financial independence. Nevertheless, the objectivity of the external evaluator can be overestimated. (Note the role of the ex- ternal Arthur Andersen firm in the 2002 Enron bankruptcy and scandal. The lure of obtaining or keeping a large contract can prompt external parties to bend the rules to
Chapter 1 • Evaluation’s Basic Purpose, Uses, and Conceptual Distinctions 29
keep the contract.) However, for programs with high visibility or cost or those sur- rounded by much controversy, an external evaluator can provide a desirable degree of autonomy from the program. External evaluators, if the search and hiring process are conducted appropriately, can also bring the specialized skills needed for a particu- lar project. In all but very large organizations, internal evaluators must be jacks-of-all- trades to permit them to address the ongoing evaluation needs of the organization. When seeking an external evaluator, however, an organization can pinpoint and seek the types of skills and expertise needed for that time-limited project.
Organizing Internal Evaluation for Maximum Effect. In recent years, evaluations conducted by people employed by the organization have grown exponentially as funders’ demands for accountability have increased. This growth is at least partly due to professional evaluators’ emphasis on building internal organizational capac- ity to conduct evaluation. (Capacity building and mainstreaming evaluation were the conference themes for the American Evaluation Association in 2000 and 2001, respectively, with the 2001 conference focusing on one of our co-authors’ themes, mainstreaming evaluation. See Leviton, [2001] and Sanders, [2002] for their pub- lished Presidential Addresses on the subjects.) We will discuss capacity building fur- ther in Chapter 9, but in this section we will discuss ways in which to structure internal evaluation to improve evaluation and the performance of the organization.
First, a comment on internal evaluators. For many years, large school districts had, and many continue to have, internal evaluation units. The economic con- straints on education have reduced the number of districts with strong internal evaluation units, but such units remain in many districts. (See, for example, Christie’s interview with Eric Barela, an internal evaluator with the Los Angeles Unified School District, Christie and Barela [2008]). In many nonprofit organiza- tions, internal evaluation capacity has increased in recent years. This growth has been spurred by United Way of America (UWA), a major funding source for many nonprofit, human service organizations, which encouraged these organizations to implement its evaluation strategy for measuring outcomes (Hendricks, Plantz, & Pritchard, 2008). Today, approximately 19,000 local agencies funded by United Way conduct internal evaluations, supplemented with training by United Way, to measure agency outcomes. Similarly, Cooperative Extensions and other organiza- tions are active in conducting internal evaluations (Lambur, 2008). State and local governments have been thrust into a more active evaluation role through federal performance-based management systems. All these efforts have prompted public and nonprofit organizations to train existing staff to, at minimum, report data on program outcomes and, often, to conduct evaluations to document those outcomes.
Given the growth in internal evaluation, it is appropriate to consider how internal evaluations can be conducted for the maximum effect. Evaluators have been writing about ways to enhance internal evaluation for some years (Chelimsky, 1994; Love, 1983, 1991; Scriven, 1975; Sonnichsen, 1987, 1999; Stufflebeam, 2002a). Probably the two most important conditions identified for successful internal evaluations are (a) active support for evaluation from top administrators within the organization and (b) clearly defined roles for internal
30 Part I • Introduction to Evaluation
evaluators. The strength of internal evaluators is their ongoing contribution to decision making within the organization. Without the active support of leaders within the organization, internal evaluators cannot fulfill that role.
Where evaluators should be located in a large organization is an area of some disagreement. Internal evaluators must be situated where they can understand organizational problems, initiate or plan evaluations to address those problems, and be in a position to frequently communicate results to the stakeholders who can use them. Some argue that internal evaluators should, therefore, be placed centrally within the organization where they can work closely with top decision makers. In this way, the internal evaluators can serve in an advisory function to top managers and are able to communicate information from a variety of evaluation studies as needed. Many, if not most, internal evaluation units are centrally located in the organization and, hence, have the potential to serve in that capacity. With proximity to top managers, the director of an internal evaluation unit can continue to demonstrate the value of evaluation to the organization.
Others (Lambur, 2008), however, have argued that internal evaluators should be dispersed among program units where they can provide useful, forma- tive evaluation for program improvement directly to people who are delivering the organization’s programs. In such positions, internal evaluators can build a more trusting relationship with program deliverers and increase the chances that the results of their evaluations will be used. Lambur, in interviews with internal evaluators in cooperative extension offices, found disadvantages to being “closely aligned with administration” (2008, p. 49). Staff who are delivering programs, such as teachers, social workers, trainers, and others, see evaluation in the central office as being more concerned with accountability and responding to federal gov- ernment demands and less concerned with improving programs. Lambur found evaluators who worked in program units were able to become closer to the pro- grams, and, as a result, they believed, knew how to conduct more useful evalua- tions. They recognized the potential for being less objective, but worked to make their evaluations more rigorous. In such positions, internal evaluators can serve in Rallis and Rossman’s role of critical friend (2000).
Patton (2008b) has also interviewed internal evaluators and has found that they face many challenges. They can be excluded from major decisions and asked to spend time on public relations functions rather than true evaluation. In addition, they do, in fact, spend much time gathering data for accountability requirements from external funding sources; this takes away time from developing relationships with administrators and people who deliver the program. Internal evaluators are often, but not always, full-time evaluators. Like many professionals in organiza- tions, they can have other responsibilities that conflict with their evaluation role.
Patton (2008b) and Lambur (2008) argue that internal evaluators face com- peting demands in evaluating for accountability and for program improvement. Both argue that the emphasis for internal evaluators should be on program improvement. Lambur writes,
“Through my personal experience [as an internal evaluator], I learned it was far more effective to promote evaluation as a tool for improving programs than helping the
Chapter 1 • Evaluation’s Basic Purpose, Uses, and Conceptual Distinctions 31
organization meet demands for accountability. If program staff view themselves as primary stakeholders for evaluation results, they are more apt to become engaged in the process of conducting high-quality evaluations. Results of such evaluations can be used first for program improvement, and then for accountability purposes.”
Those writing about the organization of internal evaluation acknowledge the difficulties an internal evaluator faces, but provide many useful suggestions. The solution, however, for an individual organization can depend on its mission and purpose. In some organizations, placing evaluation in a central location with top administrators can provide the distance from programs needed for credibility in important summative evaluations and can supply evaluators with avenues for affecting organizational learning and culture through educating key administrators about the role of evaluation. In other organizations, it can be important to place evaluators in program units where they can focus on the improvement of indi- vidual programs. In either case, internal evaluators require organizational support from top managers, mid-level managers, and supervisors. Internal evaluators can help create a true learning organization, where evaluation is looked to for valuable information to make decisions. To do so, though, requires careful planning and continuous communication and support from others in clarifying and supporting the role of evaluation in the organization.
Possible Role Combinations. Given the growth in internal evaluation capacity, considering how to combine internal and external evaluation is important. One way is to consider the purposes of evaluation. The dimensions of formative and summative evaluation can be combined with the dimensions of internal and external evaluation to form the two-by-two matrix shown in Figure 1.2. The most common roles in evaluation might be indicated by cells 1 and 4 in the matrix. For- mative evaluations are often conducted by internal evaluators, and there are clear merits in such an approach. Their knowledge of the program, its history, staff, and clients is of great value, and credibility is not nearly the problem it would be in a summative evaluation. Program personnel are often the primary audience, and the evaluator’s ongoing relationship with them can enhance the use of results in a good learning organization. Summative evaluations are probably best conducted
Internal
Formative
Summative
1 Internal
Formative
3 Internal
Summative
External
2 External
Formative
4 External
Summative FIGURE 1.2 Combination of Evaluation Roles
32 Part I • Introduction to Evaluation
by external evaluators. It is difficult, for example, to know how much credibility to attach to a Ford Motor Company evaluation that concludes that a particular Ford automobile is far better than its competitors in the same price range. The credibility accorded to an internal summative program evaluation (cell 3) in a school or nonprofit organization may be no better.
In some cases, though, funds are not available for external evaluators, or competent external evaluators cannot be identified. In many cases, summative evaluations are conducted internally and, in such cases, role combinations are possible to improve the credibility of the results. Patton (2008a) suggests using ex- ternal evaluators to review and comment on the quality of internal evaluations. In other cases, external evaluators can design critical elements of the evaluation, helping define the evaluation questions and developing evaluation designs and measures, perhaps working jointly with an internal evaluation team. Internal evaluators can then work to implement the evaluation and to develop effective means for communicating results to different stakeholder groups. Such role com- binations can save critical fiscal resources, improve internal capacity, and enhance the credibility of the results. (See, for example, Fitzpatrick’s interview with Debra Rog concerning her role as an external evaluator in a project for homeless fami- lies spanning several cities. She discusses the role of staff within each organization in helping conduct and plan the evaluation with her guidance [Fitzpatrick and Rog, 1999]). In any case, when a summative evaluation is conducted internally, man- agers within the organization need to attend to the position of the evaluators in the organization relative to the program being evaluated. They must work to ensure maximum independence and must not place evaluators in the untenable position of evaluating programs developed by their boss or colleagues.
Sonnichsen (1999) writes of the high impact that internal evaluation can have if the organization has established conditions that permit the internal evalu- ator to operate effectively. The factors that he cites as being associated with eval- uation offices that have a strong impact on the organization include operating as an independent entity, reporting to a top official, giving high rank to the head of the office, having the authority to self-initiate evaluations, making recommenda- tions and monitoring their implementation, and disseminating results widely throughout the organization. He envisions the promise of internal evaluation, writing, “The practice of internal evaluation can serve as the basis for organiza- tional learning, detecting and solving problems, acting as a self-correcting mecha- nism by stimulating debate and reflection among organizational actors, and seeking alternative solutions to persistent problems” (Sonnichsen, 1999, p. 78).