Assessment: its place and our offering
4A is the minimalist risk management framework we propose: it's an initial package which is no more than what is required or essential, but packs power. In The Risk Assessment Worm we set out how probability-impact risk analysis is the core weakness at the heart of most risk registers – and suggested alternatives.
The MODEL approach to assessment combines model-building, data and views based on expertise and experience, at the individual and group level. MODEL enables us to move beyond emotion or pure rationality, into real intelligence. The structured approach makes intuitive sense and supports better decision making.
The need and the assessor
The rationale for good assessment should be obvious; without a good assessment the wrong decisions will be taken – the decisions will be based on wrong or misleading information. The Risk Assessment Worm covered what we assess, and argued that probability and impact on a 1-5 scale are a bad choice.
The objective of MODEL is a little different; having decided what we need to assess – the probability of the Conservatives winning an outright majority in the 2015 would have been a good "non-risk" example – we want a range of tools which make that assessment better: more reliable and perhaps even more enjoyable.
Pity the assessor
Are you responsible for a "risk assessment"? Commiserations. The reason for the requirement may be unclear, along with what exactly (or even roughly) you are trying to assess. You have probably been left on your own, receiving little or no feedback on your assessment. And the probability-impact method is awful.
But even if we improve on what we are trying to assess – see above – guidance on how to improve the assessment is hardly plentiful. Let me know if I'm wrong.
The idea and the process
MODEL simply collects together the scattered ideas for improvement and brings them together in once place. The improvements include:
- Tackle bias – Keep good heuristics, but overcome belief and behavioural biases, including anchoring, availability and representativeness.
- Probability calibration – A technique whereby an individual can improve his assessment through answering questions. Best illustrated by example.
- DELPHI method – An iterative technique whereby individuals' assessments (sometimes with the rationales) are anonymously fed back to the group.
- Bayesian assessment – A method which combines expert opinion with data to produce a compound assessment; the more data the greater its weight.
- Model support – A method in which the assessment is informed by a cashflow (or other) model of the business. Particularly useful for "impact assessment".
The result is that your assessment benefits from multiple perspectives: data, experts and more. It becomes more objective and less distorted by individual biases. It has been claimed that part of the value of risk management was "getting everyone together in the same room". This gives the idea teeth and "de-risks assessment".
We also suggest that there's a natural process to go through which should bring material improvements to your assessments. It's 1-5 above.
The first two items work best at the individual level; the bias point is about making people aware of researched biases which can weaken assessment, while probability calibration gives an individual to practise and improve his assessment through a (potentially large) number of simple examples. Probability assessment is a neat technique. It can work whether the examples are realistic – specific to your domain of expertise – or not. And feedback is almost immediate.
The DELPHI technique brings the benefits of those improved individual assessment abilities together in an appropriate group environment. DELPHI enables us to benefit from the wisdom of the group, while the anonymity seeks to avoid groupthink.
Bayesian assessment objectivity and mathematical rigour to the challenge of combining data and expert (or at least group) judgement. It belongs naturally in the process after the group judgement has been formed, although the group will obviously reflect after this – the process is iterative.
The model support item is somewhat different. It uses a corporate model, primarily to focus on the impact of some scenario (although e.g. economic scenarios generators can also help with probabilities). What's the impact on pension costs if interest rates increase to 5%? Ask your model, not human risk assessors.
Leaders and risk assessment
The role of leaders in decision-making can be problematic; leaders are paid to lead, but some of their common characteristics can impair good quality decision-making. Consider three areas linking leadership and decision-making:
- Over-optimism: this affects non-leader assessors too of course. Leaders' optimism can also have benefits e.g. drive and robustness to setbacks.
- Overarching dominance: When does drive become dominance and when does dominance become abusive and dangerous?
- Agency issues: A type of conflict of interest whereby the interests of leadership and ownership become misaligned.
Expert opinion, data and models: the detail
How do these things work in practice?
The role of models
A model can have value even without any data:
- Corporate cashflow models: An insurer models annuity payments, driven by underlying contract "rules" i.e. the product definition at point of sale.
- IF-THEN effects: In corporate situations the effect of (e.g.) inflation increasing from 2% to 8% should be modelled; there is no need to speculate on the effect.
- Squash scoring: The game progresses according to scoring rules which don't require judgement. The rules allow scoring on an opponent's serve (or not).
The role of data
Data can be used on its own or to calibrate model inputs. Consider two customer retention examples from insurance:
- Cancelled from inception rate: What proportion of life insurance cases cancel in the first month, for a premium refund? Many insurers measure this – and the smarter ones use it as a leading indicator of potential agent bad debt issues – but it will not always be explicitly incorporated into a cashflow model.
- Year-1 lapse rate: What proportion of life insurance cases cancel some time in the first year? This will almost always be reflected in cashflow models; there are obvious profitability and operational implications, including provisions for clawback of provisions from any selling agent.
The role of experts
- Experts: Generally we mean people with technical or strategic insight. Sometimes these will be "industry gurus"; an outside opinion can carry more weight. Something to watch: so-called experts can be little more than good storytellers; they present a compelling or appealing narrative. But evidence for their predictive ability can be somewhat light; this applies to fund managers, authors and strategy gurus, who can be relief on to tell a good story "after the event".
- Operations: Solid operational experience can be incredibly helpful and is often underrated. When there is a strategic disaster BP? Shell? – check Adapt tales often later emerge of warnings from operational staff which were dismissed by senior management.
Operations: inviting everyone to the party
Operations in loss reduction
Economist Tim Harford in his book Adapt: Why Success Always Starts with Failure describes how the BP oil spill of 2011 was a result of senior management consistently ignoring or overruling the advice of staff "in the field". Major operational risks are naturally assessed by those with operational experience, current knowledge and involvement, rather than those in head office. Don't trust the safety of your nuclear power stations (just) to a man in Westminster.
Paul Moore is a barrister by profession and was a senior risk and compliance manager at HBOS before the bank "blew up" in the financial crisis. He believe that the "lack of voice" of those on the ground was a major contributor to what happened:
The real problem and cause of this crisis was that people were just too afraid to speak up and the balance and separation of powers was just far too weighted in favour of the CEO and their executive.Paul Moore's whistleblower statement
Poacher turned gamekeeper: bank-related fraud
The movie Catch me if you can tells the story of how Frank Abignale, a teenager in 1960s America, forged cheques to the value of millions of dollars. He was ultimately recruited into the bank fraud department of the FBI and is now a multi-millionnaire – by legal means. Frank really knew how things worked.
Many bank-related fraud cases concern bank staff – often traders. Wikipedia gives a list of trading losses over $100m, noting that for reputational reasons this list is incomplete. Traders who have also worked in the "back office" can exploit procedural or systems gaps they know of, or can advise on how to close the gaps.
Reality check: the Bernie Madoff points to lone bosses being potentially a much bigger risk than traders.
Leading indicators: insight from operations
Operations staff see at first hand what is happening "on the ground". This sometimes enables them to gains insights which are hard to obtain using other methods. More precisely, operations staff will build up practical experience of predictive metrics which may never be incorporated into risk-based assessments – unless we ask.
Technical types refer to leading indicators e.g. calories taken in and calories burned are leading indicators for weight loss / gain.
Two examples: squash and insurance
Squash example: changing the scoring rules
Squash scoring rules have been changed many times. The most recent was an attempt to increase the popularity of the game, through lobbying for its inclusion as an Olympics event and increasing television coverage. There is no evidence that a MODEL-like process was used – the results certainly weren't impressive.
But how might such a process have worked? The order of events could differ, but here's one possibility:
- Use a simple model: Compare key metrics based on a set of assumptions. For example:
- Assume two players with equal probabilities (1/2) of winning each rally.
- Calculate expected number of rallies (and points if different) per game.
- Calculate expected time to complete a game.
- Extend the above from game to match and from "expected time" to other statistics.
- Extend to when probabilities are not 1/2 (etc).
- Note that the above assumes no behavioural changes (e.g. more / less aggressive play).
- Talk to people: Players, coaches, other experts
- Without preconceptions, what are their initial views of the proposals?
- What is their response to the model results above?
- Is the assumption on no behavioural change reasonable? If not what would be?
- Any other insights and suggested next steps?
- Look for data: What's already there or could be produced?
- Consider whether data on previous changes is available.
- Consider results of previous trials – even tennis scoring produced very limited trial data.
- Do experiments behind closed doors. Produce data.
- Consult others: the general public, TV producers, Olympic committee.
Insurance example: Bayesian mortality assessment
Here's the scenario.
- An insurer takes on a number of people, each of whom has an unknown but identical probability of dying over the next year.
- To simulate random deaths in the graphs below ("a = 42" etc) we take the unknown probability as 1%.
- A substantial proportion of the business is reinsured at a rate of 1% – no profit for the reinsurer, but it's a competitive world!
The Bayesian approach updates initial beliefs with data. We show three separate initial beliefs for comparison:
- The blue curve reflects the insurer's initial and naive belief that mortality rates are uniformly distributed between 0 and 1.
- The red curve reflects the insurer's initial belief that claims may be in line with the 1% reinsurance premium.
- The green curve follow a similar logic to the red curve, but suggests the reinsurer has included a 10% profit margin, leading to an initial assumption of 0.9%.
Following the MODEL / Bayesian recipe – the account of which is slightly simplified above – our overall assessment develops as follows:
One of the nice features of this approach is that we not only get an updated best estimate of the mortality rate but also of the uncertainty surrounding the estimate. As expected, that uncertainty reduces as we get more data. For more detail see our Mortality variations article.
MODEL: beyond risk assessment
Unsurprisingly, some aspects of the MODEL framework can easily be extended to other areas where human judgement is required. In terms of the "classic" risk management process the other areas of application include "risk identification" and "risk treatment". Clearly these areas require good human judgement – the operational and expert contributions – as well as leadership.
Usually the role for data and models in identification and treatment is reduced, compared to assessment. Models and data can, however, provide information on the desirability of certain controls, especially when data from several organisations is combined; this is a valuable tool compared to relying purely on human judgement.