Risk registers: what your auditor probably won't tell you

For many companies, especially those subject to audit, risk registers are central to risk management. While advocated by "the big four", some risk experts have criticised risk registers severely: one calls them "the worm at the heart of risk management" and another suggests they can be "worse than useless".

The typical use of risk registers and, especially, the risk assessment methodology raises many issues. Often just two numbers summarise uncertainty: probability and impact. They may be shown on a two-dimensional risk heatmap, or compressed into a single number (e.g. probability * impact) or even a colour.

Aren't you already just a little uneasy? This article highlights some inconvenient truths about many risk registers and suggests solutions.

Risk registers: two surprises

Surprise #1: the flaws

The average risk register is badly flawed. Typical use of risk registers gives rise to many often unappreciated challenges. These can be grouped into five areas:

AreaThe 4ARM view
Unclear purpose and audience This is the key. If there is no clear answer to this point we should consider giving up risk registers, regulation permitting.
Methodology: avoidable flaws There are several big flaws, failure to solve which results in misleading or even dangerous results being presented.
Methodology: wrong tool for the job Once the problem is diagnosed the solution is easy. This is, at worst, a criticism of the person who explains the risk register.
Content This is calibration problem; using the best classifications and descriptions will result in better and easier management of risk.
Practical usefulness Risk registers: the claimed flaws give two examples, which may be caused by working under time pressure (or laziness).

It is my firm belief and experience that the last four areas have robust practical solutions. Whether a risk register has value in an organisation depends on many things, including the use of other tools. It's one way of answering "how do you organise, document and review the management of uncertainty?"

The article Risk registers: the claimed flaws considers these areas in more detail, listing 17 claimed flaws across the 5 areas,

Here I look at just one of the avoidable flaws: the use of probability-impact risk assessment and its consequences. See also Risk is more than events.

Surprise #2: the silence from many professionals

If the average risk regsiter is "the worm at the heart of risk management", "worse than useless" and "as much use as astrology" (Hubbard) who would tell you?

Who will speak up? Has your auditor ever mentioned the challenges associated with probability-impact risk assessment. Does he realise that risk management expert Doug Hubbard wrote a whole book – The failure of risk management – largely based on its flaws?

I get many emails about the validity of Risk Maps, Risk Scores and Risk Matrices in risk analysis. I know there are a lot of passionate proponents of these methods (and many heavily-invested vendors) but there is still no evidence that such methods improve forecasts or decisions.Doug Hubbard: Still no evidence

Is the problem that your auditor or vendor is, to quote Hubbard, "heavily-invested" at the organsational level? Is that an excuse?

Challenge your auditor. Whatever your sector, why not ask your auditor, risk manager (or actuary!) to justify probability-impact? Point out the following:

  • Private sector? Even old risk management standards such as COSO have a "get out" clause:
    This may be accomplished in two stages where an initial screening of the risks is performed using qualitative techniques followed by a more quantitative analysis of the more important risks.COSO: Risk assessment in practice
  • Public sector? The UK government's orange book likes probability-impact. But since 2004 it's only been "tweaked". A public sector expert says:
    Heatmaps don't really work at the strategic level. They try to get you to allocate a likelihood and impact to each risk. But for every risk there's a whole range of impacts... Anyone using a heat map in this way is taking a view ... and very rarely are they transparent in doing so. We need to have a simpler analysis which can be quantified. Source: Source: Trevor Llanwarne, former Government Actuary: Risk registers that work at board level
  • Voluntary sector? The UK charity commission supplies its Charities and risk management recommends event-based listing and corresponding probability-impact risk assessment, albeit boosting the importance of "impact". But the tide might be turning:
    But some areas are murkier than others. The increasingly blurry line between charities and their partners in other sectors, and emerging technological threats to charities pose risks that are not as comfortably shouldered as more traditional challenges like threats to finances or assets. Even where charities are comfortable with these risks, the options for managing them can be limited. 2015 survey: Attitudes and approaches to risk in the voluntary sector

    It is, of course, these most important strategic risks which are most poorly served by this flawed risk assessment methodology.

Probability-impact risk assessment: how does it work?

Many companies place significant reliance on the risk assessments made in risk registers. Usually some degree of assurance will be taken from the process and reporting via heatmaps. Audits of controls may use a "risk based" approach, often driven by the gross and net (of controls) risk assessments within risk registers.

It follows that if the risk assessments in risk registers are wrong, inconsistent, misleading or impossible to interpret we should be concerned.

If your firm uses probability-impact risk assessment (PIRA) here's the question: is PIRA simple or simplistic – can it easily mislead you?

Let's take a look at a "P-I matrix" which is often used to guide assessors using the PIRA methodology. Naturally not all companies will use all the features described.

Using the matrix

  1. A risk the assessor thinks of a potential event/loss over the next year.
  2. He estimates the probability of this occurring: say 10%.
  3. Using the matrix, 10% is converted to a probability rating of 2.
  4. The likely financial impact if the event occurs is estimated: say £15m.
  5. Using the matrix, £15m is converted into an impact rating of 3.
  6. The risk assessment is (probability,impact) = (2,3).
  7. This may be compressed further to 2 * 3 = 6.
  8. An amber "RAG status" may be assigned – amber in this case.

The probability-impact matrix (P-I matrix)

Impacts down the first two columns, probabilities in the bottom two rows.

The PIRA method is attractively simple. Simply add Excel functionality for a seemingly winning combination. But let's look a little deeper.

Probability-impact risk assessment: example 1 – dice

In our simple example a company has a liability which is defined at ten times the total score on two yet-to-be-rolled dice (payout in millions of pounds).

The reason for giving this example is that it illustrates how confusion can arise even where the probabilities and impacts are known.

Suppose probabilities and impacts are defined by rolling two dice. The probability is simply that of rolling the outcome. The impact is (by definition) ten times the sum of the two scores. Thus an impact of 30 = 10 * (1+2) comes from the two possibilities (1,2) and (2,1), with a total probability of 2/36.

The impacts range from 20 to 120 and are summarised as follows:

Scores (row / column 1) and impact (table body)

Score 1 / 2123456
1203040506070
2304050607080
3405060708090
45060708090100
560708090100110
6708090100110120

The probability-impact matrix (P-I matrix)

There are some minor irritations here:

  • End points muddle. An impact of £1m is "bumped up" to I = 2 in the matrix. An impact of £100m is not "bumped up" to I = 5. This is design carelessness.
  • Calibration confusion. There are three P * I values of 4. (2,2) is coloured green but (1,4) and (4,1) are coloured amber. Why?

Inconvenient truth: Although the table on the left completely characterises the "dice risk" there are several ways of summarising this. We now look at four.

Approach 1: grouping by individual impacts

Effectively there is no further summarising here, just colouring and prioritising according to the heatmap criteria.

Score 1 / 2123456
1203040506070
2304050607080
3405060708090
45060708090100
560708090100110
6708090100110120
  • The table to the left records impact
  • Each of the 36 results has a probability of 1 / 36 = 2.8% < 3%
  • So P = 1 in the matrix
  • 1 / 36 = 2.8% < 3%, so this is assigned P = 1 in the matrix
  • Since P = 1, we look down the first shaded column in the P-I matrix above
  • Impacts of more than 50 amber, smaller impacts are green
  • Makes sense, but surely no one will record 36 separate risks?

I have interpreted the "bumping up" of impacts on the edge of categories in a particular way in the above. None of this affects the central arguments.

Summary: At this point things look reasonable at an intuitive level; amber in the middle with red and green at the extremes. Now let's do some grouping.

Approach 2: grouping by common impacts

With this approach we note that there are several ways to obtain the same impact, for example 20 = (1,2) and (2,1). A full summary of each impact is:

Individual impact ProbabilityMatrix probability (P)Matrix impact (I)P * I
201/36 =   2.8%133
302/36 =   5.6%236
403/36 =   8.3%236
504/36 = 11.1%236
605/36 = 13.9%248
706/36 = 16.7%3412
805/36 = 13.9%248
904/36 = 11.1%248
1003/36 =   8.3%248
1102/36 =   5.6%2510
1201/36 =   2.8%155
Total36 / 36 = 100%   

It's getting odd: The point of greatest concern actually seems to be the most likely result (16.7%). This is not what (e.g.) an insurer means by risk.

The PIRA method issues a "call to action" via heatmaps. This commonly ignores the cost-benefit of taking action, in favour of a relatively naive approach which we might call "colouring for grown ups". The cost-benefit should be considered for all risks, but especially for those where there is a return element; in the dice example an insurer may have charged 80 for covering the risk. Returns after any losses then vary from 60 (80 - 20) down to -40 (80 - 120).

That's a good trade. In fact insurers have few such opportunities where the probabilities are known. The "control" against the -40 result would be to write a lot more such business. Other insurers would quickly realise what was going on and profit margins would be competed away. There is a theory that says the true rewards come to companies that can assess and manage subjective uncertainty, in particular probabilities that require judgement rather than just calculation.

Approach 3: grouping all impacts in a range

This approach acknowledges the range of impacts specified in the probability-impact matrix. We want to group (and sum the probabilities of) results according to whether I = 1, I = 2 etc. This comes from grouping items in the table above to produce the table immediately below:

Scores (row / column 1) and impact (table body)

Matrix Impact (I)Total Probability (%)Matrix Probability (P) P * I
1000
2030
310 / 36 = 27.8% 3 9
423 / 36 = 63.9% 4 16
53 / 36 =   8.3% 2 10

The probability-impact matrix (P-I matrix)

Here's where it gets really tricky

  • Already we have "compressed uncertainty" by grouping the 36 outcomes into the 5 "impact buckets" – this may be acceptable.
  • Did the risk assessor ever work through this process, or did they immediately think of a single scenario?
  • Was the assessor motivated by what they've read about recently, or have seen on the news> This is the so-called availability heuristic.
  • Did the assessor immediately start thinking about the "orange impact 5" event? Should he have chosen the red "3" or "4" instead? Which one?

This may seem quite abstract; you don't hear about dice on the news. What about assessing weather damage, where the scenarios are more newsworthy?

Approach 4: reporting risk as a single (P,I) point on a heatmap

The P-I matrix above is supposed to help assess a single risk. Many risk registers then plot all the risks on a heatmap. The heatmap to the right is reproduced from the COSO risk assessment guide.

Heatmap key features:

  • Risks are plotted on a likelihood (probability) and impact scale
  • The red-amber-green approach to highlighting importance
  • Other dimensions (speed and vulnerability) – no more comment here

Verdict on this assessment process

The simple reporting approach contains the seeds of greatness. There is a range of risks, including strategic risk. The multi-dimensionality of risk is recognised. The flaw is that the event-probability-impact approach compresses and may omit the most important risks.

Probability-impact risk assessment: example 2 – bad weather damage

How opaque is risk assessment?

The dice example is deliberately constructed to be transparent and shows potential problems. Real risk assessments are considerably messier. The dice example could be "unpicked" by an expert, who would appreciate the possibilities within seconds and be able to start asking pertinent questions within minutes.

The dice example benefitted from the lack of subjectivity regarding the possibilities ad their probabilities. Usually risk assessment is more complex and opaque.

Insight: Assessment of probabilities is often necessarily subjective, but there is no need to hardwire subjectivity into our methodology.

Scenarios: three types of wave

Consider three types of wave, which can be thought of as scenarios:

  1. Wind-driven ocean waves. Significant damage due to hurricanes is well known, of course, with names like "Andrew" and "Katrina" being well. Data on financial and other damage exists.
  2. Earthquake-driven tsunami waves. These have entered the general public consciousness after the Indonesian disaster in 2004 and the Japanese disaster in 2011. But over history there have been more than 25,000 (smaller) tsunamis, plus an Indian disaster in 1737, which is estimated to have killed 300,000.
  3. Asteroid-driven tsunami waves. The stuff of science fiction and films, the impact of a large asteroid could immediately cost many millions with atmospheric damage wiping out life on earth.

Important questions

  • Which scenarios were assessed? Should we omit the asteroid scenario?
  • Was data used in the assessment or was it "expert judgement"?
  • What probabilities and impacts were considered?
  • Would another assessor have produced very different results?

Three scenarios: pictures

Source: A new approach for managing operational risk

Let's draw some lessons from the dice and weather examples.

The verdict: systematically incomplete and inconsistent

Incomplete

Risk assessment using the probability-impact method – and hence risk assessment in most risk registers – is incomplete. It misses risk and understates uncertainty.

But first, let's recall that it really does miss risks. It can easily trip up even experts:

When I started at the Government Actuary's Department a few years ago, we had a risk register running to 50 pages. I looked for the top 7 risks and 3 were missing. When I mentioned this to a group of public sector risk managers I got the answer "you’ve got 4 out of 7" you’re lucky to have that many. Risk registers that work at board level – Trevor Llanwarne, former Government Actuary

It misses risks for two reasons and in two places.

First it misses risks at what is normally called the identification stage. This is probably what the former Government Actuary meant in the case of his department. The 50 pages of risks sounds like some extensive and probably over-creative brainstorming. Where were the models and strategic plans?

Several studies have looked at the "share" of risk between risk categories.

ResearchCoverageStrategic (%)Operational (%)Financial (%)
IMPACT study 2009Public companies: negative events64351
Oliver Wyman 2000Biggest share price falls61336
Though the majority of companies' efforts in enterprise risk management programs focus on financial risks, the bulk of risk exposure lies in strategic and operational risks. Impact study, 2009: first major finding

Second it compresses risks – sometimes into just a colour! – and inevitably omits some areas of risk. This article has covered this point in some detail. Compressing uncertainty really is built into the PIRA methodology. The method asks us to summarise risk into two numbers: probability and impact.

Despite the resulting simplicity, two risks are often incomparable: how do you compare (2,3) and (3,2)? Further compression – by multiplying P and I, or by using a colour – may appear to give comparability, but at what cost? But it's usually worse than this: the inconsistency is hidden.

Inconsistent

In practice it is very hard to unpack the inconsistency in risk assessment. Consider two risk assessors making an assessment:

If they are both assessing "dice" risk the probabilities are in principle known. Assuming that the calculations were carried out correctly it might be possible to uncover inconsistent risk assessments, although (even in this simple case) it is not inevitable:

  • If both assessors returned a RAG status of red that could be any assessment between 12 and 25 in the P-I matrix.
  • If both assessors returned a P-I assessment of 12 this could be 3*4 or 4*3. 4 could be 1*4, 2*2 or 4*1.
  • Whatever assessment was returned we don't know if they were even assessing the same scenario (or sub-risk) within the risk.
  • If the assessments are different they may (or may not) have been assessing different things, but the possible inconsistency is at least evident.

If they are both assessing the "bad weather" risk things are more complex. One could have been assessing wind-driven waves, while the other might have been assessing earthquake-driven tsunami waves. Without further guidance – which does not normally come in risk registers – neither approach is wrong".

Assessors could have been attempting to summarise the same scenario, or indeed the full range of uncertainty. But their subjective probability and impact assessments are likely to differ. Without substantial effort we'd never know. And few would have the tenacity to find out. But it gets worse.

Usually assessors deal with different risks and, as a result, any inconsistency becomes completely opaque. The risk register as a simple management tool has turned into a risk assessment nightmare. Are you going to something about it or hope disaster strike on someone else's watch?

Why it matters: infection across three sectors

Too often risk management focuses on "events" – bad things that might happen. "Event focus" affects all three sectors in the UK:

Recall why this is potentially a big problem:

  • What events are being "selected" from the range of possibilities? It's not just wrong but opaquely wrong.
  • The most relevant part of the "event set" may not therefore be assessed.
  • Mis-prioritisation means that attention and control may thereby be directed to the wrong risks.
  • Nature and the real world will not be forgiving of these blunders.
  • Most importantly perhaps, the "event" and "likelihood-impact" approach is likely to downplay long term strategic risks. See Sinking, fast and slow.

It matters a lot.

Putting risk assessment right: the three Cs

Given the previous material it seems obvious that we need in the following areas:

The 3 Cs of risk register improvement

  1. Clarity: Being clear on the purpose for risk assessment and overall guidance.
  2. Consistency: Ensuring that assessments will be comparable e.g. for prioritisation.
  3. Completeness: Ideally we want this, although compromise may work well.

[1] Clarity: Can we be clear on what scenario is being assessed?

In Risk modeling alternatives for risk registers (2003!) Matthew Leitch suggests 5 things that should guide our approach to risk assessment:

  1. What is it for?
  2. Is there an ultimate yardstick?
  3. Is assessment to be formal and centralised or informal and decentralised?
  4. Are "upside risks" to be considered too?
  5. Is assessment in a list or using a model?

You can design in consistency: we'll show a couple of ways of doing this below. The PIRA process has inconsistency baked in.

[2] Consistency: Can we make risk assessment consistent?

My article Better risk assessment gives no fewer than 9 alternatives to the PIRA methodology. Some are much simpler than PIRA and all are more logical and consistent. See "Alternative routes to consistency and completeness" in that document. But here are three possibilities.

My first suggestion, is to simply ask the Board to rank risks, ideally with a score from 1-10 (say) to indicates its concern. This elementary approach bypasses the PIRA pseudo-science. It captures Board concerns in almost the simplest way possible. It is clearly a simple assessment on which to build.

Due to the evident simplicity of the above, the more detailed risk assessment which COSO claims is necessary is more likely to take place.

My second suggestion, is that all risks are assessed at the same likelihood level across all risks. This means that the assessor is asked how bad the impact could be at the 1-in-10 level, say. This approach is broadly how financial firms calculate economic capital. With further refinement, it enables aggregate risk to be calculated.

We can sometimes get close to doing risk assessment without any mention of percentiles or probability. See Probability: magic without mystery.

[3] Completeness: Can we make risk assessment complete?

Here are two ways of making progress:

Where next? The risk register series

User beware. Many risk experts have warned of the common flaws in risk registers. It doesn't have to be this way. The first half of the set of articles below is generally positive, starting with how five potential audiences might make better use of risk registers. The second half warns of some really dangerous flaws.

© 2014-2017: 4A Risk Management; a trading name of Transformaction Development Limited