Defining Evaluation’s RoleIn his first months as Director of Evaluation at The Duke Endowment, Bill Bacon studied how evaluation is used at the Endowment, how it is viewed, and what changes might improve its usefulness to grantees, staff and Trustees. In the interview below, he talks about the need for evaluation and describes the role it can play in philanthropy. You joined The Duke Endowment staff in 2009 after five years as a grantmaker and evaluator at the David and Lucile Packard Foundation. What convinced you to come to the Endowment?I had a great time at Packard and learned a lot, but I was ready for new challenges. The Duke Endowment appealed for a lot of reasons, including an opportunity to contribute at a higher level and to help create an evaluation framework that cuts across all the Endowment’s programs and helps us to be the best grantmakers we can be. In philanthropy, is there a standard definition for “evaluation?”A common one is “the use of social science research methods to systematically investigate the effectiveness of social intervention programs.” That definition emphasizes the role of evaluation in determining “what works”—a program’s effectiveness or impact. But the term is also used to describe situations where you are collecting information to learn about how a program is being implemented and to help implementers improve their programs. That seems very broad. How is evaluation different from common sense information-gathering and problem-solving?The key distinction is that evaluation is systematic. It uses standard tools and practices to help discipline our decision-making and keep us objective. What’s behind the debate about whether evaluation should be for “proving” versus “improving” programs?This debate is mostly about methods. Some folks in the proving camp argue that to get really solid evidence that proves a program’s effectiveness, you should use the “gold standard” of research methods—the randomized controlled trial. Others say the real purpose of evaluation is to improve programs, and for that other methods might be more suitable and less costly. Which side do you take?I don’t take a side because I think it’s a silly argument. Clearly, evaluation can be useful for answering both types of questions. But different questions call for different sorts of evidence and there is no gold standard method that is superior in all cases. Let’s say you are implementing a program that has been shown to be effective for a particular population and you want to know how to implement the program for a different population. A couple of focus groups and some modest outcome tracking might provide exactly the kind of information you need. But evidence like that would not get the intervention onto a government-approved list of “proven programs” that are ready to be implemented at a large scale. Method choice has to be matched to the evaluation purpose. What kinds of changes are you proposing in the way the Endowment approaches evaluation?I don’t think any changes will feel dramatic. The Endowment already has a strong history of using evaluation to track and improve the effectiveness of the programs it supports, and it uses a good mix of evaluation methods matched to evaluation questions. I am developing a framework to help us take fuller advantage of the value that evaluation can provide. In particular, I’m working to ensure that we seize opportunities to apply evaluation tools and principles to every stage of the plan-implement-assess work cycle. I’m also focusing more attention on using evaluation to clarify, improve and assess our own grantmaking strategies. The Endowment's Trustees recently approved six guiding principles for evaluation at the Endowment. One of those principles is “Measure only what we will use, and use everything we measure.” What do you mean by that?I’ve been on both sides of the grantmaking relationship—as a grantseeker working at a nonprofit and as a program officer at a foundation. In both roles, I was appalled by how much time and energy could be wasted by requests for data that appeared to go into a black hole. If evaluation is to play a helpful role in promoting social betterment, it first has to do no harm. Requiring others to collect and report on data that no one will use is harmful. You’ve given us a lot to think about. If we want to learn even more, can you recommend one or two books on the subject?A seminal work on evaluation that applies both to philanthropies and nonprofits is "Utilization-focused Evaluation," by Michael Quinn Patton. There are lots of great resources online, including websites of the American Evaluation Association and McKinsey & Company’s Social Sector Office.
|
