FAB-testing: smarter decisions

Too many decisions are based only on best estimates. But like the drunkard walking down the middle of the road, the outcome may not be what we'd want.

FAB-testing is a toolset for improving core decision making in the face of uncertainty. As with our MODEL risk assessment tool, FAB-testing takes the many useful tools that already exist and improves their deployment, typically by making a better allowance for uncertainty assessment, quantification and management.

This article focuses on one main application of FAB-testing: improving RAROC as a decision tool. I am actively working on extensions.

What is FAB-testing? Why should it work?

FAB-testing is about strategic and core decisions within an organisation and using uncertainty-based techniques to make better choices. What do we mean by that?

Specifically we can use FAB-testing to look at which of two decisions A or B is preferable. One of these decisions might be "do nothing".

The foreword to Enterprise Risk Analysis for Property and Liability Insurance Companies suggests the top lesson for company management should be:

First, "experience the future" Source: David Spiller, then CEO of Guy Carpenter

FAB-testing is a tool that enables you to experience the full range of futures and to use the results to make better decisions.

A/B testing

Sometimes we can test the effectiveness of different approaches, using an experimental approach. Take driving purchases through adjusting website content and design. Someone thinks that adding postage and packing late in the process is best; mentioning too early will "put people off". Others believe in being upfront about the per item charge, while still others think postage and package should be "free" i.e. charged through higher prices? What to do?

A common, almost experimental, technique called A/B testing can be deployed, randomly assigning different people different purchase paths. The results are stored and conclusions drawn. Smart, especially as we can also target the proportions to be randomly assigned e.g. we might want 80% of purchasers to have the standard process, with 10% going to each of the other routes. Of course these percentages can be adjusted as the customer verdict(s) emerge.

A new challenge

The challenge above was rather straightforward; by the time the person has left the website you know the results: "good / bad", "yes / no", "purchase / no purchase". The percentage we were trying to estimate above captures everything. But many challenges and opportunities are more complex. Let's get specific.

An insurer faces a choice between writing two contracts, A and B. At a high level, both involve risk and return. They are not trivially comparable; A has the larger expected return, but also the bigger risk. It's impractical to run experiments in the way described above. Even if we did it seems we would have to wait years to judge the result – many life insurance contracts run for 25 years. We need a different tool.

Introducing FAB-testing

FAB-testing – future A/B testing – projects possible futures, with full allowance for uncertainty. Having captured the range of possibilities (A, B etc) we choose the decision which produces the best results, according to a pre-defined criterion (or criteria). By comparing all levels of risk, FAB-testing beats competing tools.

Why should FAB-testing work?

Traditionally people looked back over years of past data to make decisions. A/B testing shows that we can often make decisions based on more recent data. But there's nothing surprising there; so long as enough people are giving us "yes/no" verdicts then using recent data is great. But testing the future?

Insurer already have tools to tackle these decisions: profit testing, economic capital and and risk-adjusted return on capital (RORAC) are connected ideas. All three project the future – actually one "best estimate" future, with one explicit allowance for risk. Lesson: the future can be projected – FAB-testing should work.

Areas of application

Core decision making

What is an organisation's core "product(s)" which it might offer for sale? Consider an insurer using risk-adjusted return on capital to price and measure the performance of its products (life insurance, savings products etc). Core decisions include:

  • Whether to write deal A or B – incorporating FAB-testing improvements to RAROC (see below).
  • The level to set prices (rates or fees) in general – trading off margin and volume.

Surprisingly, research by McKinsey, suggests that deploying economic capital and RAROC techniques at a granular level (e.g. in transaction pricing) is less common in banks. A bank could investigate three scenarios (options):

  • A: status quo pricing
  • B: price to maximise RAROC
  • C: price using FAB-testing techniques

Yes, that's a sort of double FAB-testing.

Strategic and operational decision making

FAB-testing techniques are applicable to operational decisions: "will approach A or B maximise my network uptime and what effect might that have on customer retention?"

The Balanced Scorecard tool encourages us to postulate causal links between actions and outcomes – see the diagram on the right. Rather like an operational process, there is potential to test decision A versus decision B, allowing for uncertainty.

FAB-testing application: capital optimisation

The following graphs are based on my ongoing research into Bill Panning's work in (e.g.) Managing the Invisible. Bill adopts a similar approach to our FAB-testing, in particular allowing for the full range of future uncertainty. I have simple Excel-based tools to replicate and develop his results – and for FAB-testing.

The idea is to set capital to maximise the present value of profits, allowing for future insolvency (i.e. capital being exhausted). There are rules for whether re-capitalisation takes place. The graphs above show a general insurance example, with potential to vary the loss ratio and its volatility. The curves in each graph are:

  • Blue curve: the (basic) value of profits ("value add" or VA) allowing for the base level of capital
  • Red curve: the level of optimised capital
  • Green curve: the profitability corresponding to that optimised capital

The expected loss ratio graph divides into three broad sections:

  • The first section comprises (expected) loss ratios of up to about 70%. The business up to here is fundamentally attractive and is likely to generate underwriting profits which are more than sufficient to pay for the expenses. Small increases in the loss ratio do not mean that the business is unattractive and, optimally, more capital should be employed to ensure the business survives.
  • The second section has loss ratios of between about 70% and 76%, where business is much less attractive and "deserves" less rather than more capital.
  • The third section of loss ratios of 77%+ is where optimal capital is effectively zero. The economics of a 77%+ loss ratio and an expense ratio of 25% no longer work. The business has some shareholder value, because there may be some good years before the company becomes insolvent. Of course no regulator would allow a company to operate like this!

In the first two sections, the value add curves coincide. This suggests that, for a given loss ratio in this range, optimising capital adds little value; optimal capital varies from around 65 to 80. The change in value comes directly from the change in loss ratio; a reduced loss ratio increases the survival probability and the expected profitability for a given survival rate.

Naturally we could take this much further. Although management has some control over loss ratios there is normally an associated trade off. It could be that a lower loss ratio comes at the cost of lower volume or higher standard deviation / volatility.

The volatility graph is even more interesting; managing volatility is surely at the heart of risk management. Unsurprisingly, the graph shows that the more volatile the loss ratio is, the more capital needs to be deployed to maximise value.

Significant value is generated if we can reduce volatility, on the assumption that there are no other trade offs such as volume reduction. Most of the extra value add comes directly from volatility reduction rather than capital optimisation. Increased volatility could be beneficial, given sufficiently higher prices.

As before, there are effectively three sections of the curve: in the third the business becomes so volatile that capital can no longer cope with the threat to value.

I highly recommend the Panning paper and its methodology. Together with the Kelly Criterion, it was fundamental to the development of FAB-testing.

Improving existing tools

Here are three existing tools, with very brief suggestions for improvement – more detail for 2 and 3 in sections below.

  1. RAROC : risk-adjusted return on capital can lead to sub-optimal – even loss-making – decisions. RAROC also doesn't help "size" positions.
  2. The Kelly Criterion : although it can address the RAROC issues above, vanilla Kelly needs upgrading for unknown probabilities, risk appetite and more.
  3. The Balanced Scorecard : Often used for strategic steering, its inventor acknowledges that the BSC makes limited or no allowance for risk and uncertainty.

Who uses the above?

RAROC was invested by Bankers Trust in the late 1970s and is used especially by companies in financial services. The Kelly Criterion was first suggested in 1956. It has a rich literature and is at times the subject of emotive debate. Primarily is seems to be used by (some) gamblers, fund managers and hedge funds, reputedly including some of the biggest – who remain tight-lipped of course. The Balanced Scorecard, "invented" in the 1990s, is used by a wide range of organisations.

RAROC: the traditional solution to "shall we do A or B?"

The RAROC decision criterion maximises the ratio of two present values:

[1] The numerator: profits

The RAROC numerator is the value of expected future profits, discounted back to day-1 at a hurdle rate.

There are several different approaches to calculate such expected profit – for example traditional discounting at a high hurdle rate or a so-called market-consistent approach.

[2] The denominator: economic capital

The denominator is the capital a bank or insurer needs to set aside as an appropriate buffer against possible losses from a transaction – or indeed a business line or its operations overall.

So what's the problem?

I have often regretted the casual way the (actuarial) profession compresses the varied complexity of its affairs into a colourless present value. Frank Redington, voted the greatest British actuary ever

Not for the first time, Redington was right. Let's look at the flaws.

RAROC exposed – the flaws

Sometimes RAROC is described as "optimising profits". In fact the two aspects are:

  1. Optimise capital: the weighted average cost of capital (WACC) is minimised through selecting appropriate mix of assets, debt and equity – the funding. This is a subtle, iterative and demanding process in which financial institutions has ploughed resource and money.
  2. Maximise expected profits: Here there has been less progress. Leaving aside the practical doubts raised by McKinsey, we suggest RAROC needs theoretical adjustment. Simple examples demonstrate the flaws in maximising the numerator – indeed the flaws are easier to miss with real world complexity.

But RAROC has several flaws:

Flaw 1: RAROC-based investments can be loss making

Flaw 2: RAROC gives no guidance on the amount to invest

Flaw 3: RAROC gives no credit for real risk management

Additionally, all three of the above represent a missed opportunity for ongoing and value-adding discussions between risk and business functions.

This is the summary version of the full RAROC article – the full version gives more detail – explaining the above – and a little more maths!

RAROC enhanced – the Kelly Criterion

Background: what is the Kelly Criterion?

The Kelly Criterion (1956) is named after a US scientist, John Kelly, of Bell Labs. The criterion gives a formula to determine what proportion of wealth to risk in a sequence of positive expected value bets so as to maximize the rate of growth of wealth. The Kelly approach is positive rather than normative; it does not say that that you should maximize the rate of growth, but rather that if you want to do so (without any other constraints) the Kelly approach is the way to do so.

This is the summary version of the full RAROC article – the full version gives more detail – and a little more maths! – in the following areas:

  • Progress since the 1950s
  • Current attitudes to the Criterion
  • Implications and special cases

How the Kelly Criterion deals with the RAROC flaws

It does so as follows:

  1. Loss making investments: Kelly looks at compound returns taking into account full variability, not single period returns based on expected value.
  2. RAROC gives no guidance on the amount to invest: the proportion of capital to invest is the result or output of the Kelly Criterion.
  3. Credit for risk management: although this is not formulaic there is potential for this – see below.

The enhanced approach to RAROC also has softer benefits, suggesting mutually beneficial conversations between risk and business functions. These conversations arise naturally, since a collaborative approach is needed to tackle the above challenges. For example:

  • each of 1-3 needs an assessment of "the full range of uncertainty", including the confidence that business functions have in (e.g.) their pricing assumptions.
  • additional risk management needs creativity and its benefits depend on the level of uncertainty – a dynamic discussion.

[Aside: the Kelly Criterion does not concern itself directly with economic capital, simply with RAROC's numerator. There's no issue; we simply need to divide.]

So what's the formula?

Consider the following coin tossing scenario, which – the dependency q = 1 - p – we can abbreviate to (p,W,L):

ResultProbabilityPayoffWhat happened?
Heads = win p W We win W
Tails = lose q = 1 - p L We lose L

We know from the RAROC flaws above that investing simply if the expected "one off" gain is positive i.e. p * W - q * L > 0 is a flawed strategy. Instead Kelly says we should invest a proportion f which maximises (1 + f * W)p * (1 - f * L)q. Equivalently this maximises p * log(1 + f * W) + q * log(1 - f * L). This is a relatively simple calculus problem, the solution to which is:

The Kelly Criterion

The Kelly Criterion says that to maximize the rate of growth we should invest a proportion f = [p * W - q * L] / [W * L].

  • The numerator is our expected gain – what gamblers call the "edge"
  • The denominator is a scaling factor – this gets us beyond the expected value approach

Making it operational: FAB-testing = RAROC-Kelly-Plus

Adjustments are required to turn RAROC-Kelly into a fully working decision making tool. Five areas are described below.

Good news

Significant progress has been made in all the areas below, so that FAB-testing is a workable concept. I am not aware, however, that it has yet been deployed in practice. Deployment would inevitably and advisably us a step-by-step approach:

  • Examine current decision making methodology: Does it use RAROC and modelling?
  • Flaws in RAROC-based decisions:
    1. Negative returns: is there anything in place that stops a project with negative expected compound returns going ahead? Has this happened in practice?
    2. Sub-optimal sizing: should we have taken a different level of nominal exposure? Typically this would involve taking a share rather than the whole deal.
  • Learning the lessons: happily we can learn lessons from this approach without a full FAB-testing implementation – see below.

[1] Continuous variation

In most commercial situations gains and losses effectively follow a continuous distribution, where larger losses have smaller probabilities. The coin tossing is usually only illustrative. Suggesting that many investors would benefit from studying the Kelly Criterion, UK actuary and private investor Guy Thomas says:

It is not so much that it gives directly applicable answers, but more than it directs attention to the right questions.Source: Guy Thomas writing in Free Capital

It seems that this view is backed up strongly by Boyles asset management in this extract from their letter to shareholders.

[2] Uncapped losses

Insurers may enter into contracts which, in principle, have no finite upper limit. Necessarily (since the area under the probability curve is 1) the probabilities associated with the largest losses become very small (and are asymptotically zero). But although the economic capital associated with these is qualifiable, there is inn theory a probability – infinitesimal no doubt – that such a contract exhausts the entire capital of the company.

[3] Unknown probabilities

In most commercial applications probabilities need to be estimated, rather than being given or deduced from (e.g.) symmetry arguments. Typically the estimates comes through a combination of data analysis and expert judgement, perhaps backed by a formal Bayesian framework. Again this challenge has been tackled.

[4] Limited time horizons

Another way of putting this is that we need to incorporate risk appetite. Professor Samuelson was not the first to spot the point that most investors cannot afford to wait for the timescales or number of investment realisations that would result in the Kelly Criterion's asymptotic properties dominating other strategies.

[5] Simultaneous investments

Extensions of the Kelly Criterion exist to deal with multiple investments.

In practice we do not deploy a proportion of available capital as economic capital against a bet (opportunity, project etc), wait for the outcome then deploy another proportion against another opportunity. Instead opportunities present themselves either simultaneously or at overlapping time intervals.

We also need to think about:

  • Opportunity cost: Does taking an opportunity now mean that we may need to pass on a more favourable opportunity later?
  • Dependency: There are various types of dependency. Two investments might be:
    • directly dependent e.g. the two investments might both fail because the second required the success of the first.
    • indirectly dependent e.g. pension liabilities in two different schemes are both likely to be affected by general population longevity improvements.
    We might along the lines of financial economics, think an insurer's portfolio as a bundle of risky investments.
© 2014-2017: 4A Risk Management; a trading name of Transformaction Development Limited