Insight

Information

Article on the Brussels Seminar
How Not to Deal With Airline Overbooking: Why Equal isn’t Always Fair and Choice Mechanisms Work Better Than Command and Control
How do we transform evaluation to be more strategically relevant and timely? Adopt a cumulative learning approach.
Dealing with Complex Causality and Small Data Sets? Try a Theory Based Approach and Qualitative Comparative Analysis
Course Update: Continued Strong Reviews for Measurement and Evaluation Course for Regulators

Improvement

2013-14 Key Events

December 12, 2013

European Union (September 17-18, 2013)

The Canadian Experience Master class with Steve Montague. Steve presented a Master class to the Group of Resource Directors (GDR) of the European Commission Secretariat General on lessons from the implementation of results-based management (RBM) by the Canadian Federal Government. The overall purpose was to inform and inspire the Secretariat General’s thinking on how to improve the Commission’s own performance management and accountability framework (Activity-Based Management and Strategic Planning and Programming) and, in this context, to assess whether some elements of the RBM could be tailored to meet the Commission’s needs while matching its institutional set-up. Emphasis was on creating a shared purpose and learn from others to overcome barriers.

Upcoming 2014 Events

January 16, 2014

“A Systems Approach to Performance Planning, Monitoring and Evaluation in Complex Regulatory Environments: The Case of CFIA” For more information and registration click here.

2013 Seventeenth Annual Performance and Planning Exchange Symposium (May 13-15, 2013)

Training Session – RBM 101

A one-day, intensive course on Results-Based Management (RBM). The course covered the RBM fundamentals – from differentiating between activities, outputs, outcomes and indicators to using results for managing and reporting. Through discussion, presentations, a case study and in-class exercises, this course provided participants with a solid understanding of RBM planning and performance measurement and reporting principles and how to apply them in their environment. Participants gained a hands-on experience in building Logic Models / Outcomes Maps and Performance Measurement Frameworks.

Planning, Monitoring and Evaluating Program Impacts and ‘What works’ in Complex Environments: Emerging Canadian and International Practice

This session addressed what looks like overwhelming challenges to determining public policy and program impacts by suggesting that a non-conventional approach needs to be taken. The presenters drew on guidance from Treasury Board Secretariat, international practice and leading practitioners to suggest and demonstrate alternative approaches to determining program impacts. Analysis of need, context, theories of change and alternative measurement and evaluation approaches were explored. The session was appropriate for all levels of managers and analysts who are required to plan, monitor, evaluate, report and/or manage in complicated or complex management and public policy environments. Some basic principles for establishing expectations, policy or program value propositions, and then monitoring and evaluating initiatives in conditions of complexity were explored in both presentation and case-based work. Participants took away a set of key principles, some knowledge of emerging analytical tools and approaches, and tailored collective learning with regard to how to tell an appropriate performance story for key areas of public policy and administration.

Canadian Evaluation Society National Annual Conference 2013 (June 10, 2013)

A presentation on What works? How evaluation can cross boundaries to influence public policy and management. Evaluation has languished behind other review efforts in terms of influencing public policy and administration. This has occurred despite the fact that evaluation is designed to address fundamental questions. The session suggested that evaluation efforts needed to adopt a realistic and pragmatic approach to help public and NGO decision makers to learn and understand what works (to what extent) for whom in what conditions and why in terms of initiatives. The session demonstrated and collectively examined what we know about different policy instruments (e.g. carrots, sticks and sermons etc) applied in different areas (e.g. industrial innovation, food safety, public health and energy) and delivered via different implementation designs (e.g. single agency delivery, delivery partnerships, contribution to intermediaries etc.). The demonstration both illustrated findings which participants can use in their practice, but also demonstrated an approach to using theory-based approaches and evaluative research and studies to generatively learn about the influence of policies and programs on results and the important factors to consider when planning and implementing initiatives.

World Bank (June 28-29, 2013)

A two day logic models in development evaluations training workshop for the World Bank-Carleton University at the International Program for Development (IPDET) Evaluation Training 2013. The workshop offered an opportunity to explore logic models in more detail. It took a practical, step-by-step approach to developing traditional program logic models and offered innovative strategies for depicting the complexity involved in many of today’s development initiatives. Participants worked on a series of individual and small group exercises, had ample time to ask questions, considered plenty of practical examples, and were encouraged to share their own experiences.

American Evaluation Association 2013 Conference (October 19, 2013)

A presentation on Realistic Contribution Analysis. Approaches such as realistic evaluation (Pawson and Tilley 1997) and theory of change based approaches like contribution analysis (Mayne 1999) have typically been seen as distinctly different from each other. Furthermore, theory based approaches have sometimes been distinguished as a separate and distinct grouping from participative approaches for complicated, complex and ‘small ‘n’ evaluations (White & Phillips 2012). The presentation described the application of what might be called realist contribution analysis where the context, mechanisms and outcomes (CMO) framework which represents the essence of realist evaluation have been built into a theory of change contribution analysis approach which is developed, validated and applied as a participative process. Examples were drawn to illustrate how this approach has been applied in policy initiatives, regulatory functions, codes and standards applications and direct assistance programs. The approach was shown as a critical front end to cost-effectively scoping evaluations, a valid means to frame study designs and perhaps most importantly, as a way to engage decision-makers in unique dialogue such that evaluation use is greatly enhanced.

Community of Federal Regulators Annual National Workshop 2013 (November 4, 2013)

Steve presented together with CFIA department representatives on the CFIA Performance Measurement Reform: A System-Based Approach. This presentation discussed the implementation of changes to the department’s organization and regulations. This “modernization” includes a more unified inspection approach which recognizes that there are many participants in the food safety system who influence the ultimate outcome of protecting Canadians from preventable health risks.

Performance and Planning Exchange (December 3, 2013)

A presentation on the 2010 Public Accounts Committee report on DPRs, providing a perspective on the needs of parliamentarians. Participants were given the tools to carry out the rating process, and had coaching throughout. Rating criteria were drawn from the Guide to Preparing Public Performance Reports developed by the Public Sector Accounting Board. This learning event was a stimulating exercise, providing a renewed appreciation for the art and science of performance reporting.