Before planning the third season of a hospitality project, St. Paul’s minister of outreach suggests at a committee meeting that an evaluation might help the group to make improvements for the upcoming year. Committee members nod in agreement. One enthusiastic member suggests, “Let’s design an evaluation form participants can fill out at the end of each session.” Another member offers a dose of realism: “Great idea, but we don’t have the time or the expertise to make up a survey. Let’s just hire an expert in evaluation.” A third member opposes the very idea: “What a waste of time, we already know how it’s going and what we need to do for next year.” And a fourth member dares to voice an underlying concern of many: “Are we sure we want to ask people what they think? What if the wrong people speak up and they distort what is really happening? Won’t that possibly harm the project?”
The committee members’ responses reveal some common attitudes about evaluation: it consists of immediate feedback after an event; it is extra work added on to an already full agenda; it requires experts; and it’s challenging or possibly dangerous. Each of these attitudes reveals an element of truth about evaluation; however, they are not a complete or adequate picture of what evaluation is and what it can do for a project. Evaluation is much more. I think of evaluation as a kind of collaborative inquiry that builds on three important dimensions of strong organizations—learning, leadership, and accountability.
Standard textbooks in evaluation define the word evaluation in one of three ways: to assess the worth or merit of a particular object (Is this project worthwhile?); to assess objectives and outcomes (Did the project do what it promised?); or to gather information in order to make decisions (Based on what we know about the project, what aspects should be improved, added, or discontinued?). Each of these definitions highlights an important aspect of evaluation, but there is another dimension that I think supports all three. It might be best described as a meta-purpose—why an organization engages in assessment and decision-making. The primary motivation to engage in project evaluation is a desire to learn.
An Opportunity to Learn
But what does a project and organization seek to learn from evaluation? Quite simply, a comprehensive approach to evaluation allows us to learn something about each aspect of a project’s design and, in so doing, contributes to the refinement of the project’s rationale. Evaluation builds up a body of knowledge that can help project leaders refine activities, select appropriate resources, identify results, describe impact, and hopefully understand more fully the condition being addressed. For example, project evaluations can teach us something about the relationship between activities, resources, and impact: what resources are necessary to successfully undertake an activity, and what is the activity’s real impact on participants. In short, what we learn from evaluation can become the evidence that further substantiates the project’s claims about the condition it is addressing, as well as what the project is doing to address it.
St. Paul’s Church claims that hospitality is necessary in a large parish where people have a hard time meeting others face to face. The hospitality committee believes that newcomers need immediate assistance in finding their way into parish life. Furthermore, the church believes that mentoring relationships are the most effective means of connecting people to parish activities. As the committee looks to evaluate their efforts, they can begin with the question: What is it we most want to learn about our project? Committee members might be wondering about a number of issues: What areas of parish life are newcomers attracted to and what are they most likely to become involved in? How do existing ministries welcome newcomers and draw on their knowledge and experience? Do any newcomers take initiative to start new groups and ministries? Questions such as these illustrate what project planners might desire to learn, and evaluation is one of the most effective means by which people can learn about their work together.
Many of us are accustomed to filling out an evaluation form at the end of a meeting, event, or conference. It provides the host with immediate feedback on what people liked and disliked. Oftentimes this is valuable information and an effective means for gathering participants’ immediate feedback, but in most cases it does not constitute the full range and scope of evaluation.
When evaluation is a means to learning, it is both a formative and summative practice. Evaluation that occurs over the course of a project is generally referred to as formative evaluation; when evaluation occurs at the conclusion, it is called summative evaluation. Both types of evaluation provide helpful ways of seeing what happens in a project. Formative evaluation, for instance, gives project leaders an accurate picture of what is taking place as the project activities unfold, and may assist them in making changes over the course of the project that will insure goals are met. Summative evaluation looks back on a project and seeks to learn how and in what ways project goals were met and what kind of impact the project has had on its participants. Summative evaluation is often used for purposes of accountability or accreditation. It also provides information to strategic planners that can aid in decision-making about future efforts. Evaluation, then, can have a broad scope beyond the evaluation forms we are asked to fill out at the end of a conference.
Overcoming Resistance
If evaluation is going to do what I claim it can, it matters who in an organization is responsible for it: who leads it, who cares about it, and who uses it. Evaluation is the responsibility of organization and project leaders who help staff members, board members, and constituents understand that evaluation is important and useful. Leaders set a tone for evaluation; their attitude toward learning influences how others will receive and use evaluation information. It is no surprise that evaluation is often resisted; it can be threatening! A project, in fact, may not be living up to its claims, or securing the results it aimed for. Some people fear that evaluations will turn up negative findings that will lead to punitive measures against the project or organization. Others, on the basis of previous experience, may view evaluation as a waste of time and of little use. And some may see evaluation primarily as additional work on top of an already full set of tasks.
Evaluation may feel for some like negative judgment or being graded on their performance. A project leader may feel they have to prove something to others in order to get high marks. It can be uncomfortable to have a project judged, particularly in regards to work we care deeply about. We might feel that evaluators are looking over our shoulder and pointing out problems and failures. Because of our close involvement and commitment to the work, we may hear criticism or suggestions as negative indictments rather than as constructive proposals. And we may fail to separate what the project is and does from our personal investment and contributions to it.
Leaders can reframe evaluation to help overcome these resistances. For instance, when evaluation is truly understood as learning, the project’s weaknesses, mishaps, and failed attempts are not threats, but opportunities. Leaders can invite people to step back and reflect on the project’s goals, and identify the obstacles that are blocking success. If leaders allow people to see the project design as flexible and adaptable, evaluation opens up alternative strategies and activities. In fact, if plans do not change over the course of a multiyear project, it may be a signal that learning is not taking place. Because every element
of a successful project cannot be identified at the outset, evaluation is indispensable for answering the questions: What have we learned from the project thus far? What’s missing from what we are doing? and What needs to change?
In addition to a spirit of learning and openness, leaders can insure that evaluation is worth the time it takes. Evaluation is an activity that requires resources such as time, money, and material goods, so people must be able to see that the use of additional resources has merit for the project and organization. In essence, evaluation can create a spirit of collaborative inquiry among the project’s stakeholders. A key aspect of evaluation is determining who these stakeholders are, what they want to learn, and how the project can engage them in conversation. Again, in a well-designed project, stakeholders are people invested in the project’s rationale: they are committed to helping the organization successfully execute activities, secure needed resources, and achieve the results and impact necessary for changing a condition. Evaluation is a way to provide comprehensive information to the project’s stakeholders so that they see more clearly, understand more fully, and learn what is necessary to do the work effectively.
Making Good Projects Better
Evaluation is a means of being accountable to the organization’s mission and the project’s purposes. Obviously, few organizations have the luxury to be wasting time and resources on unnecessary or ineffective projects. Evaluation helps to build the case for a project and strengthen an organization’s commitment to sustaining it over the long term.
Project evaluation, then, has the potential to make good projects better. It increases understanding about the conditions the project seeks to address, and strengthens the claims about the activities and resources necessary for effective response. In addition, project evaluation helps us to see the project’s impact on constituents and the organization. In essence, evaluation helps to refine the rationale that provides a project with meaning, purpose, and direction.
This article was excerpted and adapted from Kathleen A. Cahalan’s forthcoming Alban Institute book, Projects That Matter: Successful Planning and Evaluation for Religious Organizations (AL 266). To order, call 1-800-486-1318, ext. 244, or visit our Web site at www.alban.org.