Evaluation Explained: A Conversation with William (Bill) Bacon

The Duke Endowment is known for its focus on evidence and evaluation. We sat down with Bill Bacon, vice president and director of evaluation, to find out why evaluation matters to the Endowment and how it shows up in grantee partnerships and initiatives.

Why is evaluation so important to The Duke Endowment? 

Our founder, James B. Duke, established The Duke Endowment to support the well-being of people and communities across the Carolinas. In the 100-plus years since the Endowment’s creation, the challenges to well-being have only become more complex. As stewards of Mr. Duke’s vision, our staff and Trustees take it as their duty to invest in programs or interventions that have the best chance of truly delivering results for the people we serve. You might think that means we invest only in programs with long track records, but that’s not the case. We also invest in innovative ideas and use evaluation to test and refine these ideas to increase their likelihood of success. 

What is the goal of evaluation in grantee programs and initiatives?

The main goal of evaluation is learning – for us, our grantees, and the broader field. Evaluation can determine whether an intervention had a positive effect on participants. It can identify which components of a new intervention are most effective or make an established intervention more successful. It can also help us recognize challenges and what might need to change when the results aren’t what we anticipated. 

As we learn, we are helping to build knowledge for the field. That’s important, too, because smaller foundations may lack the resources to support large-scale, rigorous evaluations, so a foundation like The Duke Endowment can contribute important knowledge to the field by filling that role. 

Are there certain types of evaluations that you prefer your grantees use?

The most important thing is to choose an evaluation that will help you learn what you need to learn, given the stage of development of your program. Many interventions evolve over years or even decades, moving from idea to planning to implementation to (sometimes) broader scale. Different types of evaluation – implementation, outcomes, and impact – are appropriate depending on the stage of that evolution. The right evaluation can help you learn what you need to learn as you plan, implement and eventually, if successful, expand your program. 

What is an implementation evaluation?

As the name suggests, an implementation evaluation is typically used early in a program’s evolution to determine how well the program is being implemented and what additional program components, training or other resources might be needed to help reach program goals. For example, suppose you have an idea about reducing frequent emergency room visits among a particular population. Your idea may be completely new, or it may be based on an existing practice you have replicated or adapted to suit your target group. 

An implementation evaluation would help you better articulate the key ingredients necessary to make your approach work, such as hiring and training the right people or creating standards to deliver the program consistently with quality. An implementation evaluation can help define those key ingredients and assess whether they are in place. You can ultimately save yourself time and resources if you really study and understand what it takes to implement the program well before getting too serious about measuring outcomes. 

When might it be appropriate to measure a program’s outcomes? 

Outcomes are the beneficial changes to participants, including measurable changes in knowledge, attitudes, behavior, or health or well-being status. We usually differentiate outcomes from outputs, which are just the direct results of program activities such as the number of workshops held or people served. Outputs are essential for reporting on grant activities, but we love to see measures of outcomes whenever it’s practical. When it’s feasible to measure outcomes really depends on the program. Sometimes it is appropriate to measure them from the outset. A program that trains clinicians in a new technique, for example, could use a knowledge test to assess learning and a standardized observation protocol to evaluate skill development. In the Endowment’s Summer Literacy Initiative, we conducted several years of outcome monitoring – measuring changes in children’s reading skills from the beginning to the end of their participation – before we even considered a more technically demanding impact evaluation. That gave us time to refine the intervention and ensure we could implement it consistently across multiple sites.

What is involved in an impact evaluation? 

An impact evaluation is typically conducted only after a program has matured enough to be implemented with quality and consistency and offers a pretty good signal that participants are benefiting. Going beyond monitoring outcomes, an impact evaluation is designed to establish whether a particular intervention caused beneficial outcomes for participants. It does this by measuring a program’s outcomes against a carefully chosen comparison group. Going back to our Summer Literacy example, that program tracked outcomes only among kids in the program for several years, which was useful. But as we began thinking about scaling the program across many more sites, we wanted greater confidence that the program was causing benefits that wouldn’t have occurred without the program. After all, it’s possible that kids would improve their reading ability over the summer even if they were not in our summer reading program. 

By comparing a group that is receiving an intervention to a control group that is not, we are able to better understand whether observed outcomes can reasonably be attributed to the intervention. A well-conducted impact evaluation generally requires an outside professional evaluator, a study group large enough to detect meaningful effects and an intervention that has been well defined and implemented with quality and consistency. 

We’ve heard that the Endowment favors randomized controlled trials (RCTs) for impact evaluations. Is that true? 

It’s true that we consider RCTs to be the ​“gold standard” for determining whether an intervention works, but they are not always appropriate. They are best suited to mature programs that require strong evidence of causal impact – not usually for smaller or early-stage efforts. An RCT may be wasteful overkill when a grantee is implementing an ​“evidence-based program” – one that is already supported by compelling causal evidence. In that case, evaluation dollars may be better used to assess whether the program is being implemented well and with fidelity to the tested model. 

How do you work with grantees to support their evaluation efforts?

We support our grantees with the connections and financial resources to undertake well-designed evaluations. Our team is delighted to connect grantees with evaluators, implementation science specialists and other consultants to help support interventions throughout their evolution. We also consider evaluation to be a legitimate program expense that can be included in grant budgets. Finally, through initiatives such as Evidence Insights, we highlight how our grantees are using evaluation to build evidence for their organizations and for the broader field.

Does the need to evaluate ever end? 

Evaluation is never one-and-done – not for our grantees and not for us. The need to learn more about what’s effective never ends. The field of evaluation is constantly evolving, so we continue to study emerging practices and methods and update our approaches and standards. We’ll never stop learning about evaluation, the interventions we’re funding, and what works to make life better for everyone in the Carolinas.