How do we transform evaluation to be more strategically relevant and timely? Adopt a cumulative learning approach.
March 14, 2014
Recent evaluation leadership meetings in federal circles have suggested that the function needs to become more timely while also becoming more strategically useful to decision-makers. In my evaluation practice and teaching I have come to the conclusion that a greater emphasis on cumulative learning vis avis program archetypes is a good part of the answer to this challenge. For a recent brief discussion on cumulative learning see http://vimeo.com/84228311 featuring realist evaluation co-founder Ray Pawson. (The answer to his question btw re: who coined the term carrots, sticks and sermons can be debated, but to me was made prominent by Carleton’s Bruce Doern in his writings in public administration several decades ago. The term was directly referenced in the evaluation context by M. L. Bemelmans-Videc, Ray C. Rist, and Evert Vedung in their book of same name published in 1998.)
Presentations at the 2010 CES Victoria conference (Building on our Strengths – http://evaluationcanada.ca/distribution/20100504_montague_steve.pdf) 2011 AEA presentation in California (http://comm.eval.org/communities/resources/viewdocument/?DocumentKey=f68c48e7-a170-44d3-b2d8-689ce430d343), the 2012 Halifax CES conference (http://evaluationcanada.ca/distribution/20120516_montague_steve.pdf) and a workshop at the 2013 CES Toronto conference – outlined some of the basic thinking about how such an approach might be applied for Canada. In the last instance – I have even gone as far as to suggest a sort of streamlined approach to spur such cumulative learning (https://www.pmn.net/wp-content/uploads/Checklist-for-Context-and-Policy-Instruments.pdf).
Over the past few years we have developed the approach for specific carrots (e.g. Grants + Contributions) sticks (e.g. Administrative Monetary Penalties) and sermons (e.g. voluntary and compulsory standards and codes development, other types of information/education and policy programs) archetypes. This has already provided us with significant cumulative learning regarding what works for whom in what conditions and why. This knowledge in turn represents an asset and ‘leg-up’ to conducting essentially any future evaluation of these types of programs. Readers who would like to learn more and/or join in this movement are invited to contact us at info@pmn.net and/or to stay tuned for presentations at future events!
Steve Montague, Partner PMN