The reference case for management reserves

May 23, 2017

Risk management and Earned Value practitioners, and a range of standards, advocate the inclusion of contingencies in the project baseline to compensate for defined risk events. The contingency may (should) include an appropriate allowance for variability in the estimates modelled using Monte Carlo or similar; these are the ‘known unknowns’.  They also advocate creating a management reserve that should be held outside of the project baseline, but within the overall budget to protect the performing organisation from the effects of ‘unknown unknowns’.  Following these guidelines, the components of a typical project budget are shown below.

PMBOK® Guide Figure 7-8

The calculations of contingency reserves should be incorporated into an effective estimating process to determine an appropriate cost estimate for the project[1]. The application of appropriate tools and techniques supported by skilled judgement can arrive at a predictable cost estimate which in turn becomes the cost baseline once the project is approved. The included contingencies are held within the project and are accessed by the project management team through normal risk management processes. In summary, good cost estimating[2] is a well understood (if not always well executed) practice, that combines art and science, and includes the calculation of appropriate contingencies. Setting an appropriate management reserve is an altogether different problem.

 

Setting a realistic management reserve

Management reserves are an amount of money held outside of the project baseline to ‘protect the performing organisation’ against unexpected cost overruns. The reserves should be designed to compensate for two primary factors.  The first are genuine ‘black swans’ the other is estimating errors (including underestimating the levels of contingency needed).

The definition of a ‘black swan’ event is a significant unpredicted and unpredictable event[3].  In his book of the same name, N.N. Taleb defines ‘Black Swans’ as having three distinct characteristics: they are unexpected and unpredictable outliers, they have extreme impacts, and they appear obvious after they have happened. The primary defence against ‘black swans’ is organisational resilience rather than budget allowances but there is nothing wrong with including an allowance for these impacts.

Estimating errors leading to a low-cost baseline, on the other hand, are both normal and predictable; there are several different drivers for this phenomenon most innate to the human condition. The factors leading to the routine underestimating of costs and delivery times, and the over estimating of benefits to be realised, can be explained in terms of optimism bias and strategic misrepresentation.  The resulting inaccurate estimates of project costs, benefits, and other impacts are a major source of uncertainty in project management – the occurrence is predictable and normal, the degree of error is the unknown variable leading to risk.

The way to manage this component of the management reserves is through the application of reference class forecasting which enhances the accuracy of the budget estimates by basing forecasts on actual performance in a reference class of comparable projects. This approach bypasses both optimism bias and strategic misrepresentation.

Reference class forecasting is based on theories of decision-making in situations of uncertainty and promises more accuracy in forecasts by taking an ‘outside view’ of the projects being estimated. Conventional estimating takes an ‘inside view’ based on the elements of the project being estimated – the project team assesses the elements that make up the project and determine a cost. This ‘inside’ process is essential, but on its own insufficient to achieve a realistic budget. The ‘outside’ view adds to the base estimate based on knowledge about the actual performance of a reference class of comparable projects and resolves to a percentage markup to be added to the estimated price to arrive at a realistic budget.  This addition should be used to assess the value of the project (with a corresponding discounting of benefits) during the selection/investment decision making processes[4], and logically should be held in management reserves.

Overcoming bias by simply hoping for an improvement in the estimating practice is not an effective strategy!  Prof. Bent Flyvbjerg’s 2006 paper ‘From Nobel Prize to Project Management: Getting Risks Right[5]’ looked at 70 years of data.  He found: Forecasts of cost, demand, and other impacts of planned projects have remained constantly and remarkably inaccurate for decades. No improvement in forecasting accuracy seems to have taken place, despite all claims of improved forecasting models, better data, etc.  For transportation infrastructure projects, inaccuracy in cost forecasts in constant prices is on average 44.7% for rail, 33.8% for bridges and tunnels, and 20.4% for roads.

The consistency of the error and the bias towards significant underestimating of costs (and a corresponding over estimate of benefits) suggest the root causes of the inaccuracies are psychological and political rather than technical – technical errors should average towards ‘zero’ (plusses balancing out minuses) and should improve over time as industry becomes more capable, whereas there is no imperative for psychological or political factors to change:

  • Psychological explanations can account for inaccuracy in terms of optimism bias; that is, a cognitive predisposition found with most people to judge future events in a more positive light than is warranted by actual experience[6].
  • Political factors can explain inaccuracy in terms of strategic misrepresentation. When forecasting the outcomes of projects, managers deliberately and strategically overestimate benefits and underestimate costs in order to increase the likelihood that their project will gain approval and funding either ahead of competitors in a portfolio assessment process or by avoiding being perceived as ‘too expensive’ in a public forum – this tendency particularly affects mega-projects such as bids for hosting Olympic Games.

 

Optimism Bias

Reference class forecasting was originally developed to compensate for the type of cognitive bias that Kahneman and Tversky found in their work on decision-making under uncertainty, which won Kahneman the 2002 Nobel Prize in economics[7]. They demonstrated that:

  • Errors of judgment are often systematic and predictable rather than random.
  • Many errors of judgment are shared by experts and laypeople alike.
  • The errors remain compelling even when one is fully aware of their nature.

Because awareness of a perceptual or cognitive bias does not by itself produce a more accurate perception of reality, any corrective process needs to allow for this.

 

Strategic Misrepresentation

When strategic misrepresentation is the main cause of inaccuracy, differences between estimated and actual costs and benefits are created by political and organisational pressures, typically to have a business case approved or a project accepted. Reference class forecasting will still improve accuracy, but the managers and estimators may not be interested in this outcome because the inaccuracy is deliberate. Biased forecasts serve their strategic purpose and overrides their commitment to accuracy and truth; the application of reference class forecasting needs strong support from the organisation’s overall governance functions.

 

Applying Reference Class Forecasting

Reference class forecasting does not try to forecast specific uncertain events that will affect a particular project, but instead places the project in a statistical distribution of outcomes from the class of reference projects.  For any particular project it requires the following three steps:

  1. Identification of a relevant reference class of past, similar projects. The reference class must be broad enough to be statistically meaningful, but narrow enough to be truly comparable with the specific project – good data is essential.
  2. Establishing a probability distribution for the selected reference class. This requires access to credible, empirical data for a sufficient number of projects within the reference class to make statistically meaningful conclusions.
  3. Comparing the specific project with the reference class distribution, in order to establish the most likely outcome for the specific project.

The UK government (Dept. of Treasury) were early users of reference class forecasting and continue its practice.  A study in 2002 by Mott MacDonald for Treasury found over the previous 20 years on government projects the average works duration was underestimated by 17%, CAPEX was underestimated by 47%, and OPEX was underestimated by 41%.  There was also a small shortfall in benefits realised.

 

This study fed into the updating of the Treasury’s ‘Green Book’ in 2003, which is still the standard reference in this area. The Treasury’s Supplementary Green Book Guidance: Optimism Bias[8] provides the recommended range of markups with a requirement for the ‘upper bound’ to be used in the first instance by project or program assessors.

These are very large markups to shift from an estimate to a likely cost and are related to the UK government’s estimating (ie, the client’s view), not the final contractors’ estimates – errors of this size would bankrupt most contractors.  However, Gartner and most other authorities routinely state project and programs overrun costs and time estimates (particularly internal projects and programs) and the reported ‘failure rates’ and overruns have remained relatively stable over extended periods.

 

Conclusion

Organisations can choose to treat each of their project failures as a ‘unique one-off’ occurrence (another manifestation of optimism bias) or learn from the past and develop their own framework for reference class forecasting. The markups don’t need to be included in the cost baseline (the project’s estimates are their estimates and they should attempt to deliver as promised); but they should be included in assessment process for approving projects and the management reserves held outside of the baseline to protect the organisation from the effects of both optimism bias and strategic misrepresentation.  As systems, and particularly business cases, improve the reference class adjustments should reduce but they are never likely to reduce to zero, optimism is an innate characteristic of most people and political pressures are a normal part of business.

If this post has sparked your interest, I recommend exploring the UK information to develop a process that works in your organisation: http://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-governent

_____________________

[1] For more on risk assessment see: http://www.mosaicprojects.com.au/WhitePapers/WP1015_Risk_Assessment.pdf

[2] For more on cost estimating see: http://www.mosaicprojects.com.au/WhitePapers/WP1051_Cost_Estimating.pdf

[3] For more on ‘black swans’ see: http://mosaicprojects.wordpress.com/2011/02/11/black-swan-risks/

[4] For more on portfolio management see: http://www.mosaicprojects.com.au/WhitePapers/WP1017_Portfolios.pdf

[5] Project Management Journal, August 2006.

[6] For more on the effects of bias see: http://www.mosaicprojects.com.au/WhitePapers/WP1069_Bias.pdf

[7] Kahneman, D. (1994). New challenges to the rationality assumption. Journal of Institutional and Theoretical
Economics, 150, 18–36.

[8] Green Book documents can be downloaded from: http://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-governent


Two exceptional workshops at PGCS 2017 – 1st May

April 20, 2017


PGCS 2017 is offering workshops by Dr. Lynda Bourne and Dr. Keith Joiner in Canberra on Monday 1st May. Both offer a unique international viewpoint on very different aspects of project management.

My (Lynda’s) workshop ‘Leading Successful Teams’ focuses on collaborative teams which are key to success in any business activity. The most effective teams consist of individuals who can work independently on their own tasks, but also recognise the need to work collaboratively with other team members toward the activity’s goal and the organization’s success.

The leader of the team contributes significantly to team success through inspiring all team members to work together to achieve this goal, but must also intervene to reduce conflict and to motivate team members to continue to work collaboratively.

This session will focus on the needs of first-time team managers and will consist some theory, and a little practice, on the following topics:

–  Motivation

–  Delegation

–  Giving feedback

–  Resolving conflict.

This full day workshop is based on my Master’s course I’ve been running at EAN University in South America for the last 5 years and offers exceptional value at $450 (catering and GST included)

 

Keith’s workshop ‘methods for test design and analysis prescribed in U.S. Industry & Defence’ will introduce and illustrate the new methods in test design and analysis are, and how they are used to:

– screen for significant design factors;

– model design factors;

– screen for operational factors;

– model operational factors; and

– where equipment is taken off-the-shelf, improve the efficiency of validating performance.

Participants will use an instructional toy system and study several example uses to reinforce how the methods work.

This half day workshop is great value at $330 (Afternoon tea and GST included)

Both workshops offer exceptional value and are open to everyone – you do not need to attend the PGCS symposium to enjoy these process…… For more information and bookings see:  http://www.pgcs.org.au/program/workshops/


New on the Web #16

April 9, 2017
Binnacle: designed to reduce magnetic deviation so a compass remained accurate.

Binnacle: designed to reduce compass error!

We have been busy updating our websites with Posts, White Papers, and Articles. Some of the more interesting uploaded in the last few weeks include:

These links are directly related to stakeholder engagement and communication.  A full indexed listing of all of our White Papers, Conference papers, books and articles can be found in our PM Knowledge Index.


New on the Web #15

March 16, 2017
Binnacle: designed to reduce magnetic deviation so a compass remained accurate.

Binnacle: designed to reduce compass error!

We have been busy updating our websites with Posts, White Papers, and Articles. Some of the more interesting uploaded in the last few weeks include:

These links are directly related to stakeholder engagement and communication.  A full indexed listing of all of our White Papers, Conference papers, books and articles can be found in our PM Knowledge Index.


How the chair can make a meeting ineffective

March 7, 2017

The chair of any meeting has a unique ability to destroy the value of the meeting!

Eight of the key ways to reduce the meeting’s value are:

  1. Playing favourites. Bad chairs tend to shut down some attendees whilst allowing others they see as politically important to occupy most of the speaking time. The outcome from this behaviour tends to be poor decision-making; bad chairs don’t care. Their interest is to stay the good books of the people they see as politically important.
  2. Changing the rules. Bad chairs keep the rules to themselves and change the rules when it suits them. They don’t give advice on what preparation attendees need to make or advise how the meeting will be conducted. While this trait may appear to appear to be a gambit to leave the chair in control, in reality it means the meeting is likely to be less than useful.
  3. Showing bias. When there is a vigorous debate between various groups in the meeting a bad chair will obviously be supporting one side.  Good chairs remain neutral whilst they may feel strongly about subject their primary function is to ensure the meeting reaches a consensus, not that the meeting reaches a decision that they predetermine as being optimum (although they need to be part of the consensus).
  4. Failing to define its purpose. Bad chairs do not define a clear objective for the meeting, fail to set priorities, and don’t circulate an agreed agenda. Good chairs define the purpose of every meeting with crystal clarity so attendees can come prepared and stay focused.
  5. Losing control. The hallmarks of a bad chair during the meeting include running over time, getting off track, get rattled, and allowing discussion to descend into personal arguments. Good facilitators keep their hands firmly on the reins consistently and politely guiding discussion back to the purpose of the meeting.
  6. Failing to communicate. Bad chairs tend to display no sense of appreciation for the points made by contributors to the discussion and tend to ignore many of the attendees. Good chairs are great communicators remember everybody’s name, include newcomers, and are excellent at active listening and summarising points to ensure everybody has a clear understanding of the current state discussion[1].
  7. Failing to make decisions. Deadlocks happen in most meetings, bad chairs cannot solve them. A good chair will either take a vote, extend discussion for a set (limited) period, set up a working party, or call an extraordinary meeting to deal with the item later; any of these options are better than allowing the meeting to waffle on allowing tension and confusion to grow.
  8. Failing to engage with meeting participants outside of the meeting. Bad chairs are missing in action, too busy to be involved with the delegates other than during the meeting. Good chairs recognise the meeting is part of a continuing process that requires responsive input and support between meetings.

Meetings are an expensive resource often costing thousands of dollars an hour to run. If you are the chair of the meeting, or are responsible for calling a meeting, you need to ensure the meeting is managed effectively to maximise the opportunity for success.  This is important for every type of meeting from a short team ‘stand-up’ through to company board meetings – the further up the hierarchy the greater the cost of ineffective meetings. Unfortunately ‘bad chairs’ seem to be common at all levels; the idea for this post came from an article by Kath Walters in the AICD March 2017 magazine focused on the behaviour of dysfunctional boards of directors.

Recognising poor performance is one thing, doing something about it is another; for more on managing effective meetings see: http://www.mosaicprojects.com.au/WhitePapers/WP1075_Meetings.pdf

___________________

[1] For more on active listening see: http://www.mosaicprojects.com.au/WhitePapers/WP1012_Active_Listening.pdf


The Yin and Yang of Integrated Data Systems

December 13, 2016

Integrated project management information systems (PMIS) are becoming more common and more sophisticated ranging from ‘web portals’ that hold project data through to the potential for fully integrated design and construction management using BIM[1].  The benefits from using these systems can be as much as 20% on complex construction projects using BIM.

pmisThe advantages of this type of information storage and retrieval system include:

  • Ready access to data when needed via PDAs and ‘tablets’ significantly reducing the need for ‘push’ communication and the existence of ‘redundant data’[2].
  • One place to look for information with indexing and cross-referencing to minimise the potential for missed information.
  • Audit trails and systems to ensure only the latest version of any document is available.
  • Cross-linking of data in different documents and formats to assist with configuration management, requirements traceability and change control.
  • Controls on who can ‘see’ the data, access the data and edit the data.
  • Workflow functions to remind people of their next job, list open actions, record actual progress, etc[3].
  • A range of built-in functions to validate data and avoid ‘clashes’, including locking or ‘freezing’ parts of the data set when that information has been moved into ‘work’.

These benefits are significant and a well-designed system reduces errors and enhances productivity leading to reduced costs, but the ‘yin’ of well-designed PMIS comes with a ‘yang’!

People increasingly tend to believe information produced from a computer system, this is true of ‘Facebook’, Wikipedia and flows through to more sophisticated systems. There also seems to be a steady reduction in the ability of younger people in particular to critically analyse information; in short if it comes from the computer many people will assume it is correct. Add to this the ability of many of the more sophisticated PMIS tools to transpose and transfer information between different parts of the systems automatically or semiautomatically and there is a potential for many of the benefits outlined above to be undermined by poor data. This issue has been identified for decades and has the acronym GIGO – garbage in, garbage out.

The question posed in this blog is how many projects and project support organisations (PMOs, etc.) consider or actively implement effective data traceability.  Failed audits, overruns from scope oversights, and uninformed or ill-informed decision-making are just a few of the consequences project teams suffer from if they do not have full traceability of their project management data. This issue exists in any information processing system from basic schedule updating, through monthly reporting to the most sophisticated, integrated PMIS. If you cannot rely on the source data, no amount of processing will improve the situation! And to be able to rely on data, you need to be able to trace it back to its source.

tracabilityTraceability is defined as ‘the ability to trace the location, history and use of each data element’. This sounds simple but in reality can be very challenging, and the results of poor visibility can be devastating to a project. Some of the key questions to ask are:

  • Where did this data or these actuals come from?
  • What is the authorizing document and when did it get signed/approved?
  • Has everyone approved the change request or action item?

Traceability does not happen by accident! Project management information systems have to be designed with traceability as a key element in each of its aspects.  As information comes into the system the author or the origin of the information has to be recorded (preferably automatically). Depending on the nature of the information it may need to be quarantined until appropriate checks have been carried out and/or approvals have been obtained and then there needs to be traceability of any subsequent changes. The foundation of traceability is the combination of processes (people) and data management.

Therefore, the ‘yang’ of a sophisticated integrated project management information systems is that as the systems become more integrated and sophisticated people will come to rely on the information provided and ‘trust it’ whilst the source and veracity of the data used becomes less obvious.

Resolving this is partly process and partly people. The Chartered Institute of Building (CIOB) has produced the Time and Cost Management Contract Suite 2015 focused on complex construction projects using BIM.  This contract defines a number of key support roles (largely independent of the parties) focused on managing the information flows into and out of the system to ensure its accuracy and validity. Similar roles and responsibilities are essential in any effective PMIS.

My latest post on the PMI ‘Voices blog’, From Data to Wisdom: Creating & Managing Knowledge highlights the importance of data as the underpinning of all reporting and communication.  So the question is, how much focus does your project team or PMO put on ensuring the data it is using is timely, complete, accurate and traceable?

_________________

[1] BIM = Building Information Modelling, see: http://www.mosaicprojects.com.au/WhitePapers/WP1082_BIM_Levels.pdf

[2] For more on planning project communication see: http://www.mosaicprojects.com.au/Mag_Articles/ESEI-09-communication-planning.pdf

[3] A discussion on how these capabilities can enhance project controls is at: https://mosaicprojects.wordpress.com/2016/11/26/the-future-of-project-controls/


New on the Web #14

November 29, 2016
Binnacle: designed to reduce magnetic deviation so a compass remained accurate.

Binnacle: designed to reduce compass error!

We have been busy updating our websites with Posts, White Papers, and Articles. Some of the more interesting uploaded in the last few weeks include:

These links are directly related to stakeholder engagement and communication.  A full indexed listing of all of our White Papers, Conference papers, books and articles can be found in our PM Knowledge Index.