Fall 2018 Updates

October 17, 2018

1. In the works – A knowledge take-up and use performance measurement tool. For an update – ask us.

2. The Fall 2018 European Evaluation Society – in Thessaloniki Greece: Great location, great content. For more on the content ask us.

3. Can you meaningfully target results in public administration? (hint: Yes you  can – see response to ‘Does Deliverology Matter?’)

A Time For Key Principles: What the UN has learned

December 21, 2017

S. Montague, December 2017

This year in the circles we have been working in there has been a strong emphasis on implementing the Canadian Results Policy of 2016. The policy has a few new tracking (tagging) of data elements to it and some structural requirements that are slightly new – but otherwise it can be seen as similar to past managing for results and results based management initiatives instigated by the Government of Canada and others over the past two decades (and in some senses much longer). The Policy essentially focusses on the idea that public administration – including the political levels – should focus on the  results of policies and programs – rather than input levels, activities and outputs. This is of course easier said than done anywhere – let alone in large bureaucracies.

Some of us have been recently involved in both consulting assignments and some capacity building efforts over the past year which have provided us with a firsthand perspective on  the issues in terms of the system’s ability to ‘gear-up’ to manage for results. Rather than list the issues, the gaps and the factors influencing them which exist at many levels in the system, it might be useful to instead look for inspiration from the results of other related efforts to review and assess the state of play in terms of a public administration  ‘results’ movement.

The United Nations  took on the idea of results based management (RBM) around the turn of the twenty-first century. Canadians like Perrin and Mayne clearly influenced some of the early thinking and their ideas helped shape some of the early guidance.  There have been a number of reviews conducted of RBM  over the past decade and a half – all of which have been somewhat critical of the key elements of the  movement – the idea that RBM goes beyond technical aspects of defining performance, measuring / monitoring / evaluating and reporting on results. The idea is that management would use results and performance information to manage. Unfortunately a recent survey of UN managers suggests that the vast majority did not see evidence based decision making more than ‘occasionally’. Even worse – only a small fraction believed performance analysis was done ‘honestly’.

Having done assignments for two UN organizations over the past three years and having taught a number of UN (and other) staff and managers at things like IPDET over the past decade, I have some idea as to what has been the problem.  It seems that in some cases – the implementation of RBM focused on technical aspects like having indicators and making them Specific, Measureable, Achievable, Realistic and Time driven (SMART). The problem is that this in some cases seems to have driven people to measure what is easy and controllable – not what is right and useful for management. Additionally – the ethos surrounding RBM has in some cases been strongly focused on accountability rather than learning and improvement. This seems to have driven out a spirit of continuous improvement in many cases and replaced it with fear of failure. Some would say that this is most natural to bureaucracies – and especially the ‘political’ or ‘socio-political’  bureaucracies of government organizations. They do not accept failure well – since to admit failure can cause voter support to drop – especially in competitive democratic systems – and therefore administrators  work on denying failures. To deny failure is in turn to deny learning and improvement.

So how do we break this seemingly intractable cycle documented by a succession of analysts and reviewers? It won’t be easy – but a recent UN study by its Joint Inspection Unit suggests a slightly different approach to RBM. They emphasize vision, thinking about how and why results occur and systems thinking – before discussing  SMART indicators, monitoring and evaluation.  I include an excerpt from the draft final report here.

Source: United Nations System-Wide Results-Based Management 
Analysis of Stage of Development and Outcomes Achieved in Managing 
For Achieving Results – Draft November 2017

For many of us, this signals a return to an emphasis on evaluative and deeper thinking about results. The inference is that the typical public enterprise results story is complicated and complex and requires an understanding of context,  systems and a chain of results – including some theory or explanation of why the chain of results is expected to occur. Note the principles also ask for the consideration of assumptions and factors affecting results (Principle 3 above).  If we take this kind of thinking the implications for all of us are potentially profound – from who should be truly ‘leading’ the development and refinement of results stories, to how performance profiles are constructed, what should be included in Departmental Performance Reports, to how  monitoring and measurement should be done. It should  also cause us to pause and reflect on what competence and capacity in RBM looks like. These principles suggest that it should in essence look for the development of key mental models.  It likely means a rethink of not just our epistemologies for RBM – but our fundamental ethos and logos.

If ethos and logos aren’t your thing – and you want to make one single change based on this work – then consider this: The ‘R’ in UN Principle 4  for RBM stands for relevant.  Even that small beginning change in orientation may help.

All the best to all of us attempting to promote evidence use in management, results based management and evaluative thinking in the new year!

2016 in Review – and we thought last year was in turmoil!

December 13, 2016

I guess I am an analyst at heart. I always find it interesting to look back at what one observed at a certain point in time and then to see what has happened since.  This year I am going to do that explicitly by revisiting observations we/I made about last year which were really sort of admonitions about the future.  I am then reporting on my perceived community progress since then.

1. New Government – Emphasis on Results

2015 Observation:  This year’s Canadian Fall election brought a new government with a clear and direct interest in ‘real change’, transparency and evidence-based policy and programs. This tone at the top cannot be understated – and the interest generated at all levels of public and related enterprise is palpable. A door has opened.

2016 Update:  There has been significant activity on this file since the new Canadian Government came into power. We may, however be facing a situation of rushing in some areas where we really need to think things through. The results and delivery methodology introduced to the Federal Government in February 2016 arguably needs a fair bit of adjustment to suit Canadian Federal Government circumstances. Using evidence to support decision-making means much more than developing more scorecards or simple minded delivery plans. After all – people have been trying to get this right for decades – world-wide. The good news is that Canada may have some of the answers in its results logic based approach – especially for the hard to measure areas. What we need to do is to make sure all relevant wisdom is brought to the table – even if this causes some minor delays in implementation.  (See Community of Practice discussion below.)

2.  Results Chain as a ‘Useful’ Basis for Measurement and Evaluation

2015 Observation:  The notion of using a results chain as a ‘useful’ basis for evaluation has emerged directly.  Most recently see Mayne, J. (2015). “Useful Theory of Change Models.” Canadian Journal of Program Evaluation 30(2): 119-142 and Mayne, J. and N. Johnson (2015). “Using Theories of Change in the Agriculture for Nutrition and Health CGIAR Research Program.” Evaluation 21(4): 407-428. These articles propose an approach we have been advocating for some time and create part of the conditions for real progress and maybe even transformation in our function.

2016 Update:  Experience continues to suggest that a results chain or results logic approach is a key to getting performance planning, measurement and evaluation ‘right’. At PPX we started a community of practice for science related areas (http://ppx.ca/en/exchange-communities-practice/) and it seems clear that the thoughtful, results logic-based approach developed across several Canadian provinces and some federal agencies over the past seven years (and de facto longer) is proving useful in helping to plan, analyze, report and manage science based policies, programs and initiatives.  There appears to be potential in a number of other areas as well.

3.  Program Delivery Arrangements Affect Results

2015 Observation:  In item 2 above, the notion that results chains apply to the set-up and delivery arrangements for a program every bit as much as the actual execution of the program has helped many of us to explain how and why certain programs continue to fail or persist in ‘bending’ in implementation. There are “Many a slip twixt cup and lip” as Shakespeare said. In many cases we can see that the delivery arrangements just don’t fit the value proposition (Theory of Change). For example, annual funding renewals and heightened renewal uncertainty in program areas requiring long term, certain and steady funding wreak havoc on management and often creates perverse effects. There are scores of other examples. For now suffice to say that there is some strong potential for cumulative learning here. If you have time – consider what we said about the Economic Action Plan and whether it actually created economic stimulus through its infrastructure funding in 2009.  http://www.pmn.net/wp-content/uploads/Why-Cash-for-Clunkers-Works.pdf.  Hopefully we can avoid similar folly ahead.

2016 Update:  Last year we used an old example regarding infrastructure spending (though it may be relevant again) on the Economic Action Plan to show how faulty implementation negated the benefit of a stimulus initiative. This year I have used a case example on Shared Services derived from a newspaper investigative analysis to suggest the same thing. See http://ottawacitizen.com/news/national/built-to-crash-the-ugly-sputtering-beginning-of-shared-services-and-how-politics-conspired-against-it. (Email me if you would like to see my presentation on this.)

Going forward – we will need to augment results and delivery plans with a recognition that the delivery governance, machinery and design have their own ‘theory’ to them (with their own critical success factors) and that we should include consideration of this in any analysis of performance. The new climate change agreement signed by federal and provincial actors is a case in point.  Given its flexibility, It is likely to represent an excellent learning opportunity to analyze governance, design and implementation as well as the relative merits of carbon taxation, vs. cap and trade, vs. regulatory or volunteer approaches. The fact is that groups such as the Auditor General of Ontario are already second-guessing the results logic in so many words. (See http://www.auditor.on.ca/en/content/annualreports/arreports/en16/v1_302en16.pdf. For many of us the review goes far beyond the bounds of what constitutes an audit.) The report does this without the benefit of what might have been a more fulsome examination of historical comparative evidence on the various elements of the implementation and theory of change in effect for Ontario’s cap and trade system.  In my view we need to do such analyses in a more systematic and structured fashion – as evaluations.

4. Archetypal Program Theories Inform Analysis

2015 Observation:  Following from the above – the notion that there can be archetypes of results logic or program theories for different types of programs is starting to ‘take’. We have seen that many funding programs operate on similar results logic and principles, as do many regulatory and information/advisory programs. With that in mid it stands to reason that there is much to be gained in learning what works (to what extent) with whom under what conditions and why – by starting from a synthesis of what we know about the key contextual factors that have allowed programs or policies showing similar patterns to work. This also allows us to develop appropriate monitoring, measurement and evaluation schemes, as well as to set appropriate benchmarks.

2016 Update:  Stay tuned re: the work of our science community of practice, as well as other work we have been doing vis a vis regulatory, corporate, policy and other types of carrot, stick, sermon or other policy instrument types. We are starting to see an extensive amount of common ground across groups using similar archetypes. The results may be transformational in terms of being able to cost-effectively review performance.

5.  Monitoring and Evaluation – a Collective Learning Approach

2015 Observation:  Finally – points 1-4, along with our immensely advanced recent ability to communicate quickly and easily with one another obviate the emerging view that monitoring and evaluation are part of a learning approach and that this works best as a team sport. Specialists and generalists, line managers and corporate reviewers, delivery agents and users/clients work best when they work together to understand the need, the relevance of a given policy or program or set of policies and programs to a given situation and context, how things are intended to work (and why we think they will), how they actually work in specific sets of conditions, with/for whom, why and what can be done about it.

2016 Update:  The team sport aspect of review work has begun to take hold in more areas over the past year. In our practice it is now typically standard to hold workshops on results logic among key stakeholders – at the beginning, during and sometimes after major PM+E projects and exercises. The question of “What works with/for whom (to what extent) in what conditions and why?” is emerging as a key to engaging diverse groups in a collective learning journey which can complement or sometimes even replace the adversarial ‘accountability’ modus operandi typical of conventional audit and many cost-cutting review approaches. (The realist approach which focusses on this question was probably one of the two most prominent approaches showcased at the European Evaluation Society conference in October.)  At the end of the day – our experience has been that we can improve negative findings acceptance if we have engaged key groups on a collective learning journey (and we have substantively reviewed evidence vs. the results logic of delivery arrangements and the enabling environment as well as outcomes.)

So we have had some evolutions on the themes expressed last year – however for me they all still ring true – some with more urgency than others. We look forward to working with many of you to help advance the file in these exciting and yet extremely challenging times in Canada and in our world.

Impact Pathways for Science Initiatives Released

November 15, 2016

A well received report describing science impact pathways was recently noted by an NRC official at the November 7th Science-based Organization public forum.  The report Study of Large Scale Research Infrastructure Impact Assessment was co-authored by PMN partner Steve Montague and associate Gretchen Jordan.  The full report has just been released for public access.  Email info@pmn.net to receive a copy of the full report .

Presentation by Steve Montague and Bridgette Dillon at the 12th European Evaluation Society Biennial Conference 2016

October 5, 2016

The presentation by Steve Montague and Bridgette Dillon on Developing Useful Programme Theories for Complex Interventions formed a logical flow and reinforced each other so as to provide the perspectives of an experienced commissioner of evaluations and that of an experienced practitioner leading evaluation study teams. Audience members were encouraged to interactively participate with the presenters at key periods throughout the session. For the full presentation click here:  Developing Useful Programme Theories for Complex Interventions.

Results Chain Approach Validated by Scottish Paper on Research Uptake and Impact

October 7, 2015

The results chain approach and content found in most of the articles published on the PMN web site – dating back decades (see https://www.pmn.net/wp-content/uploads/PMN-Results.pdf for a retrospective results chain illustration of the acceptance of reach and results chains – as contributed to by PMN work) continue to gain acceptance.  Sarah Morton of the University of Edinburgh shows what she has called  a ‘research grounded contribution framework’,  using a model that S. Montague and PMN have been using for some time now (see link above) – as evidenced by the references in the PMN library.

For a link to a free PDF of the Morton study see:  Morton, S. (2015). “Progressing research impact assessment: A ‘contributions’ approach.” Research Evaluation available at http://rev.oxfordjournals.org/content/early/2015/08/14/reseval.rvv016.full.pdf+html – check out Figure 1 and then Tables 1-3. They should look familiar!

Reach And How It Can Be Used to Improve Theories of Change

December 7, 2012

For more on reach and how it can be used to improve theories of change see recently published article by S. Montague and N. Porteous titled The Case for Including Reach as a Key Element of Program Theory in Evaluation and Program Planning. The article is available at http://dx.doi.org/10.1016/j.evalprogplan.2012.03.005.

Theory-based Approaches presentation well received at Canadian Evaluation Society’s Annual Learning Event

February 29, 2012

The presentation entitled Theory-based Approaches for Practical Evaluation (https://www.pmn.net/wp-content/uploads/Theory-based-Approaches-for-Practical-Evaluation.pdf) played to standing room only at the Canadian Evaluation Society’s Annual Learning Event on February 21, 2012.

For reproductions of the presentation by Steve Montague and by Joanne Roulston and Eric Seraphim of the Canadian Food Inspection Agency see the above hotlink to our library. Presentations by Treasury Board Secretariat representative Brian Moosang and Human Resources and Skills Development Canada representative Terry Colpitts should be available through CES.

Steve Montague made Fellow of the Canadian Evaluation Society

May 26, 2011

We are pleased to note that PMN partner and co-founder Steve Montague was made a Fellow of the Canadian Evaluation Society at the Society’s National Conference in May 2011. This is the highest award which can be conveyed on Canadian Evaluation Society members.

 Cllick here for details: http://www.evaluationcanada.ca/site.cgi?s=1&ss=1&_lang=EN&num=01354

Rave Reviews

February 16, 2011

Rave Reviews For Regulatory Performance Measurement and Evaluation Facilitated by Steve Montague

Recent Canada School offerings of the course Regulatory Performance Measurement and Evaluation have been receiving rave reviews from attendees. The following table is an excerpt from the actual course evaluation report produced by the Canada School of the Public Service for the November 2010 offering of the course.

Source: Canada School of Public Service, November 30, 2010.

Future offerings of this course can be seen at http://www.csps-efpc.gc.ca/cat/det-eng.asp?courseno=R003. For other course offerings by Steve Montague – and / or customized course offerings – email info@pmn.net.