Archive for the ‘Insight’ Category

Article on the Brussels Seminar

How Not to Deal With Airline Overbooking: Why Equal isn’t Always Fair and Choice Mechanisms Work Better Than Command and Control

by Steve Montague

My wife and I recently celebrated our 30th wedding anniversary with a trip to Europe. Everything went well until the last day when we went to Schiphol Airport to check in at KLM – Air France. We were shown as ‘standbys’ on our tickets. This was news to us. We had booked the trip four months in advance with KLM directly. At that time we understood that seats came with the booking. Apparently we were wrong.

Most of us who travel frequently know that airlines overbook flights with the anticipation that in most cases they will be able to easily reduce or eliminate the overage because a proportion of passengers will not show up. In this case KLM miscalculated by two people. Guess who those two people were? (Let me set aside the fact that to us, we had booked directly, booked early – and then gotten ourselves to first in line for standby when we were informed of the problem. This was all for naught as KLM chose to keep large travel groups together by assigning all available standby seats to people in bigger groups. More on this below.)

You see we were told that we had been randomly selected to be bumped to ‘stand by’ due to the overbooking. Random selection means an equal chance of selection across all members of a given cohort – in this case, passengers who had booked seats on this flight. That sounds fair enough – assuming airlines are insistent on overbooking to ensure a full load on each trip. (That is a policy that clearly places productivity over quality in terms of values – but that is for another discussion.)

The problem is what happened after the random selection. As noted, despite getting ourselves quickly registered as first standbys, the lead agent decided that anyone in groups should not be separated – therefore she bypassed us and assigned seats to those who were apparently associated ‘with groups’ ahead of us. So despite our likely having paid the highest ticket price, booked early and gotten ourselves to the head of the standby queue earliest in terms of registering with an agent, the allocation of the standby seats shifted from random to ‘purposive’ – as we would say in sampling language – using an apparent decision rule of “don’t separate groups”. In the application of this decision rule I suppose it at least united us in our misery.

Our loss was eventually (after much ‘discussion’ I will skip for my purposes here) compensated by a re-booking at KLM expense on a flight home via KLM and then another carrier in a different city 6 hours later. We also got food vouchers and a significant sum of Euros for our trouble. (In full disclosure, we did get a choice in this compensation – either a certain sum of Euros or a certain greater sum – about 25% higher ‘value’ – in future travel vouchers with KLM. We chose the cash.)

Aside from my skepticism that the allocation of standbys was in fact random in this case (for example, it seems highly improbable that all random standby assignments would be allocated to just 3 groups of people out of a total number of about 400 seats on the plane. There were two among a group of student travelers, four from a group of senior travelers and of course my wife and myself as the ‘third group’. There was also notably no allocation of random standbys to first class, business class or economy ‘plus’ class seats.)

So what is wrong with this picture? Here are the two fundamental principles or truisms that KLM got wrong with regard to service or more broadly, decision implementation.

Equality vs. fairness

Allocating from a random assignment where supposedly each passenger would have an equal chance of getting bumped may sound ‘fair’ out of context, but it runs completely contrary to the implied economic and social model and contract of modern travel. Seats are differentiated by price. The higher one pays – the higher is one’s expectation of seat security. Since none of our eight random selections came from any of the higher cost seats one suspects that the selection was in fact a stratified random sample. In other words it was randomized within a basic lower priced seating choice. So I suspect that KLM did not tell us the truth in saying that the selection was random. If that is the case – then why not stratify by the actual fares paid and on top of that then inform those paying the lowest fares that at these prices the purchasers may be subject to a standby bump? That would be fair.

Command and control vs. choice

More importantly, the full control over who was to get the seats they paid for and who was to get bumped was made by KLM according to their own criteria as to who was worthy of seats and who was not. This centrally controlled method of ‘non’ seat allocation or ‘seat rationing’ caused unnecessary grief – not just for my wife and I, feeling unfairly selected ‘out’ but also for other passengers – as the final stand by ‘count down’ was made and a great deal of anxiety was clearly expressed by many people and their co-passengers. I’m sure this was also felt by KLM staff as well as others prior to the final rulings being made by the ‘queen bee’ of seat election – KLM’s lead agent at the gate.

How could this have been done with less burden (time and effort), less confusion, less anxiety, more fairness and almost certainly less cost to KLM? Simple – they could have offered people money to take a later flight. I have observed this tactic used several times in North America, and my guess is that the amount of the offer could have been as little as half (or even less) the amount that KLM actually ended up paying us in compensation. This is because the marginal cost of a delayed flight varies quite a bit among passengers and that out of an approximate number of 400 people it is likely that the amount that two people (the 0.5 % of the whole cohort most easily bribed to delay their flight plans) would have taken to be bumped would have been a fraction of what my wife and I thought was fair compensation. Of course the real bonus is that this would have been a decision made collectively – not selectively – and therefore would have been satisfying, to the extent possible, the free choice of those concerned.

So what is the moral of our story? Service providers and policymakers at many levels should recognize that equal is not always fair and that allowing a diverse set of people to make less coerced choices almost always produces a better solution than relying on a command and control function from a centralized source of decision making. This has been articulated many times by mainly North American authors such as James Surowiecki in The Wisdom Of Crowds and by the authors of Nudge – Thaler and Sunstein.

So now to my cautiously grand over-generalization of this anecdote… (so take it for what it is worth…) There is much I love about Europe. It’s social conversations appear to be more frequent, more thoughtful, fairness oriented and wide ranging than many of our North American ones in terms of policies, programs and services. But if this small incident is in any way indicative of the European approach to service or policy implementation – or by implication – economic policy execution – as compared to that of North America, then as a North American investor I will likely keep my money (and perhaps even my airline travel choices) close to home at present. One needs to recognize market transactions as market transactions. Central ‘regulation’ needs to be there for fundamental safety, security, health and some complex economic circumstances – but when it comes to uncomplicated price-consumption choices – it’s best to let people collectively decide through a market mechanism.

How do we transform evaluation to be more strategically relevant and timely? Adopt a cumulative learning approach.

Recent evaluation leadership meetings in federal circles have suggested that the function needs to become more timely while also becoming more strategically useful to decision-makers.  In my evaluation practice and teaching I have come to the conclusion that a greater emphasis on cumulative learning vis avis program archetypes is a good part of the answer to this challenge. For a recent brief discussion on cumulative learning see featuring realist evaluation co-founder Ray Pawson. (The answer to his question btw re: who coined the term carrots, sticks and sermons can be debated, but to me was made prominent by Carleton’s Bruce Doern in his writings in public administration several decades ago. The term was directly referenced in the evaluation context by  M. L. Bemelmans-Videc, Ray C. Rist, and Evert Vedung in their book of same name published in 1998.)

Presentations at the 2010 CES Victoria conference (Building on our Strengths –  2011 AEA presentation in California (, the  2012 Halifax CES conference ( and a workshop at the 2013 CES Toronto conference – outlined some of  the basic thinking about how such an approach might be applied for Canada. In the last instance – I have even gone as far as to suggest a sort of streamlined approach to spur such cumulative learning  (

Over the past few years we have developed the approach for specific carrots (e.g. Grants + Contributions) sticks (e.g. Administrative Monetary Penalties)  and sermons (e.g. voluntary and compulsory standards and codes development, other types of information/education and policy programs) archetypes. This has already provided us with significant cumulative learning regarding what works for whom in what conditions and why. This knowledge in turn represents an asset and ‘leg-up’ to conducting essentially any future evaluation of these types of programs. Readers who would like to learn more and/or join in this movement are invited to contact us at and/or to stay tuned for presentations at future events!

Steve Montague, Partner PMN


Dealing with Complex Causality and Small Data Sets? Try a Theory Based Approach and Qualitative Comparative Analysis

Theory based approaches (see lend themselves to a broad range of applications, including situations of high complexity and small numbers of cases. These are situations where both conceptually and practically, conventional statistical analysis just doesn’t fit (Befani, Lederman and Sager 2007). European evaluators (Befani, Lederman and Sager 2007, Sager and Lederman 2012) have recently suggested that Qualitative Comparative Analysis (QCA) can be combined with theory-based approaches to provide at least a partial answer to this problem. The following deck presents a simple illustration of how realistic evaluation and a QCA approach might be combined (click here for the deck  At PMN we have begun to use this approach in our current evaluation practice. We find it straight-forward, potentially cost-effective and compelling. For more information contact

Course Update: Continued Strong Reviews for Measurement and Evaluation Course for Regulators

A recent delivery of R003 Performance Measurement and Evaluation for Regulators was March 24th and 25th at the Canada School of Public Service. We note that facilitator – PMN partner, Steve Montague continued to get exceptional reviews – in spite of the fact that the course is attracting a much broader group than regulators. The facilitator ratings out of ‘5’ for effectiveness, creating an environment conducive to learning and for knowledge averaged 4.8, 4.8 and 4.92 respectively. Some participant comments included:


– “It was very informative and beneficial.”

– “Steve is an extremely passionate and informed individual. This course could not have a better instructor.”

– “Outstanding! Knowledgable, friendly and accessible. Excellent pace.”

– “Facilitator was excellent – very engaging and extremely knowledgeable. The course itself was also engaging and I believe will be applicable to my day to day activities in the future.”

-“The instructor’s enthusiasm and knowledge of the subject matter were very helpful in making this course enjoyable and productive.”

Update: Our 2009 Prediction Re: Stimulus Funding Comes True

Globe and Mail November 24th The Great Infrastructure Boom That Wasn’t Tim Kiladze

Report on Business reporter Tim Kiladze noted that analysts had suggested that investors were far too optimistic about the extent to which public stimulus funding would boost the fortunes and bottom lines of the construction industry. Kiladze quotes a market analyst as follows: “Very few meaningful stimulus-related infrastructure projects were launched in 2009. In fact, somewhat ironically, some infrastructure spending was actually delayed … as [provincial, state and local] governments awaited funding from the federal stimulus coffers,” noted CIBC World Markets analyst Paul Lechem.

For the full article see:

For a systems explanation of why this ‘irony’ occured – predicting such a result before the fact – see the following article from August 2009. Why Cash For Clunkers Works Better Than Company Bail-Outs or Infrastructure Spending: Reach, Roles and Relationships Count in Our Results Theories

Explaining The Funding Flow ‘Success’ of Some Stimulus Strategies Over Others – Steve Montague, August 2009

In the following article Steve Montague suggests why some policy instruments get implemented more quickly and consistently than others. See: Why Cash For Clunkers Works Better Than Company Bail-Outs or Infrastructure Spending: Reach, Roles and Relationships Count in Our Results Theories.

Reach in Logic Models and Spheres of Influence

A European conference related to performance planning and reporting in the public sector has noted two important contributions made by PMN and S Montague in a paper titled ‘Three Spheres of Performance Governance: Spanning The Boundaries From Single-Organizational Focus Towards A Partnership Network’. The two important insights were noted in a paper by Petri Uusikylä and Ville Valovirta prepared for the European Group of Public Administration (EGPA) 2004 Annual Conference – Four Months After: Administering The New Europe, September 1-4, 2004, Ljubljana, Slovenia.

Uusikylä and Valovirta conclude:

One of the important insights made by the Canadian public management developers… The reach describes the groups of beneficiaries, clients, users and recipients at whom the outputs are targeted. According to Montague (1998) the logic models which do not make a reference to who and where the action is taking place, suffer from three key problems. First, they lack the sensitivity to the impacts on different participant groups. There is also a great potential to confuse outputs and outcomes – the problem which we noticed already as regards the logical framework approach. Third, the account of the trade-off between the reach and the results remains often underdeveloped. Too ambitious results are often expected since the customer reach has not been identified.”

“Another great insight made by the Canadians is the division of the performance objectives and measures into three spheres of influence (Montague 2000). The first one is the operational sphere over which the managers have a direct control. The second sphere is one of behavioural change over which the managers have a direct control. Here the question is how our actions modify people’s actions. The third sphere the environment of indirect influence, affected by the change of behaviour. What the model does is that it impersonates the rather mechanistic approaches of performance management by employing the concept of reach and by acknowledging the different degrees of influence in the three spheres.” The online version of this article can be found at: