Dealing with Complex Causality and Small Data Sets? Try a Theory Based Approach and Qualitative Comparative Analysis

March 12, 2012

Theory based approaches (see http://www.pmn.net/wp-content/uploads/Theory-based-Approaches-for-Practical-Evaluation.pdf) lend themselves to a broad range of applications, including situations of high complexity and small numbers of cases. These are situations where both conceptually and practically, conventional statistical analysis just doesn’t fit (Befani, Lederman and Sager 2007). European evaluators (Befani, Lederman and Sager 2007, Sager and Lederman 2012) have recently suggested that Qualitative Comparative Analysis (QCA) can be combined with theory-based approaches to provide at least a partial answer to this problem. The following deck presents a simple illustration of how realistic evaluation and a QCA approach might be combined (click here for the deck http://www.pmn.net/wp-content/uploads/Realist-Synthesis-and-Qualitative-Comparative-Analysis.pdf.  At PMN we have begun to use this approach in our current evaluation practice. We find it straight-forward, potentially cost-effective and compelling. For more information contact steve.montague@pmn.net.

Course Update: Continued Strong Reviews for Measurement and Evaluation Course for Regulators

June 15, 2011

A recent delivery of R003 Performance Measurement and Evaluation for Regulators was March 24th and 25th at the Canada School of Public Service. We note that facilitator – PMN partner, Steve Montague continued to get exceptional reviews – in spite of the fact that the course is attracting a much broader group than regulators. The facilitator ratings out of ‘5’ for effectiveness, creating an environment conducive to learning and for knowledge averaged 4.8, 4.8 and 4.92 respectively. Some participant comments included:

“Excellent!”

- “It was very informative and beneficial.”

- “Steve is an extremely passionate and informed individual. This course could not have a better instructor.”

- “Outstanding! Knowledgable, friendly and accessible. Excellent pace.”

- “Facilitator was excellent – very engaging and extremely knowledgeable. The course itself was also engaging and I believe will be applicable to my day to day activities in the future.”

-”The instructor’s enthusiasm and knowledge of the subject matter were very helpful in making this course enjoyable and productive.”

Update: Our 2009 Prediction Re: Stimulus Funding Comes True

February 15, 2011

Globe and Mail November 24th The Great Infrastructure Boom That Wasn’t Tim Kiladze

Report on Business reporter Tim Kiladze noted that analysts had suggested that investors were far too optimistic about the extent to which public stimulus funding would boost the fortunes and bottom lines of the construction industry. Kiladze quotes a market analyst as follows: “Very few meaningful stimulus-related infrastructure projects were launched in 2009. In fact, somewhat ironically, some infrastructure spending was actually delayed … as [provincial, state and local] governments awaited funding from the federal stimulus coffers,” noted CIBC World Markets analyst Paul Lechem.

For the full article see:

http://investdb4.theglobeandmail.com/servlet/story/GAM.20101124.GIINFRASTRUCTURE1124ATL/GIStory/

For a systems explanation of why this ‘irony’ occured – predicting such a result before the fact – see the following article from August 2009. Why Cash For Clunkers Works Better Than Company Bail-Outs or Infrastructure Spending: Reach, Roles and Relationships Count in Our Results Theories

Explaining The Funding Flow ‘Success’ of Some Stimulus Strategies Over Others – Steve Montague, August 2009

February 14, 2011

In the following article Steve Montague suggests why some policy instruments get implemented more quickly and consistently than others. See: Why Cash For Clunkers Works Better Than Company Bail-Outs or Infrastructure Spending: Reach, Roles and Relationships Count in Our Results Theories.

Reach in Logic Models and Spheres of Influence

January 26, 2011

A European conference related to performance planning and reporting in the public sector has noted two important contributions made by PMN and S Montague in a paper titled ‘Three Spheres of Performance Governance: Spanning The Boundaries From Single-Organizational Focus Towards A Partnership Network’. The two important insights were noted in a paper by Petri Uusikylä and Ville Valovirta prepared for the European Group of Public Administration (EGPA) 2004 Annual Conference – Four Months After: Administering The New Europe, September 1-4, 2004, Ljubljana, Slovenia.

Uusikylä and Valovirta conclude:

One of the important insights made by the Canadian public management developers… The reach describes the groups of beneficiaries, clients, users and recipients at whom the outputs are targeted. According to Montague (1998) the logic models which do not make a reference to who and where the action is taking place, suffer from three key problems. First, they lack the sensitivity to the impacts on different participant groups. There is also a great potential to confuse outputs and outcomes – the problem which we noticed already as regards the logical framework approach. Third, the account of the trade-off between the reach and the results remains often underdeveloped. Too ambitious results are often expected since the customer reach has not been identified.”

“Another great insight made by the Canadians is the division of the performance objectives and measures into three spheres of influence (Montague 2000). The first one is the operational sphere over which the managers have a direct control. The second sphere is one of behavioural change over which the managers have a direct control. Here the question is how our actions modify people’s actions. The third sphere the environment of indirect influence, affected by the change of behaviour. What the model does is that it impersonates the rather mechanistic approaches of performance management by employing the concept of reach and by acknowledging the different degrees of influence in the three spheres.” The online version of this article can be found at: http://evi.sagepub.com/content/13/4/399