A Few Doubts about the Stress Tests

[This is the talk on the Bank of England’s stress tests given by Kevin Dowd to the Adam Smith Institute, March 17 2016. The second edition of Kevin’s No Stress:  The Flaws in the Bank of England’s Stress Testing Programme will be published soon by the Adam Smith Institute.]

[This draft: June 5 2016.]

Good afternoon everyone. 

I’d like to begin by thanking you all for coming, and to thank my friends at ASI for the invitation to speak. 

My topic today is the Bank of England’s stress tests.  These remind me of Tweedledee’s definition of logic: 

If it was so, it might be; and if it were so, it would be; but as it isn't, it AIN’T!

You might describe the stress tests as a perfect example of policymaking by fairy tale. 

Not so much the appliance of science, but the appliance of make-believe to policymaking. 

We base policy on a single scenario that we admit will almost certainly never occur!

Lewis Carroll would have had a field day with that one!

Its almost as bad as believing six impossible things before breakfast.

1. Why the stress tests matter

So why do the stress tests matter? 

They matter because we are concerned about the financial health of the banking system. 

The Bank of England’s narrative is nothing if not clear …

From the Bank’s perspective, the glass is full. 

From my perspective, the glass is empty – not half full, not half empty  … bone dry.

The Bank’s narrative is that the stress tests show the resilience of the banking system

That the banking system has successfully recapitalized post-crisis.

Governor Carney put this nicely at the December 1st press conference:

the [UK banking system] has built capital steadily since the crisis … It is a resilient system, you can see it through the stress tests. … The long walk to higher capital is over.

My narrative is almost the exact opposite, but based on the Bank’s own data: 

The stress tests do not reveal banks’ strength, because that strength isn’t there. For example, tests based on the ratio of CET1 capital to Risk Weighted Assets are useless because RWA is a pretend number, and the banks’ capital rebuild is based on these pretend numbers.

To assess bank strength we should focus on the leverage ratio instead. Traditionally, the leverage ratio is the ratio of core capital to total assets – not Risk-Weighted Assets. 

In fact, banks’ leverage ratios are still low – by my best estimate, less than 3.7% on average. 

This is VERY low.

The long walk to higher capital has barely begun. 

2. The fatal errors in the stress tests

Turning to the tests, these are not unreliable. They are subject to a string of fatal errors. 

Let me just run through a few of these. 

The Bank of England’s stress tests miss banks’ weakness because they use extremely low pass standards. 

They rely on a single adverse macro scenario, but the chances of that occurring are very low. 

Think about it – no single scenario can possibly give you confidence that you are safe in the face of all the other scenarios you didn’t consider: it’s just not possible.

There are also other problems:

Stress testing creates new systemic risks that are invisible to everyone’s risk management systems: it creates icebergs that our ships’ radar systems cannot see, and you might even say that this is a design feature of the system.

There is a kind of Goodhart’s Law at work by which risk models break down when used for risk control purposes.

Central bank stress tests are also subject a number of credibility issues.

One purpose is to determine how strong the banking system is.

Another purpose is to promote confidence in the banking system.

But these objectives will clash if the banking system is weak.

It’s OK to promote confidence, but whether that confidence is justified or not is another matter.

So what happens if the central bank conducts a stress test and discovers that the banking system is really weak?

Can you imagine it admitting that?

Listen up everyone: better run and get your money out!

All that taxpayer money we threw at the banking system didn’t work!

No, I didn’t think so either.

Instead, the stress tests always give the same answer everywhere – the unrelenting message is always the same - the banking system is just fine … even when it later turns out it wasn’t.

Stress testing also has a positively dire track record. 

This track record shows that the underlying methodology is totally unreliable.

Real rocket science put a man on the moon, but stress test ‘rocket science’ has produced a string of disasters.

Stress tests in the US told us that Fannie Mae and Freddie Mac had a failure probability of zero; they were unsinkable, like the Titanic. 

They then sank spectacularly a few years later. 

Regulatory stress tests told us that the Icelandic banking system was safe, then the Irish banking system, then the Cypriot one, and then the Greek one. 

All these national banking systems failed shortly after the stress tests had certified them as safe. 

Relying on these exercises is a form of Russian roulette: most of the time you will be OK, once in a while you won’t.

With one near exception, I am not aware of a single case where a stress test correctly identified a risk build-up to which the authorities responded in the right way, by taking suitable counter measures in advance to defuse the problem.

This near exception was of course Northern Rock. 

In this case, the UK tripartite authorities produced a scenario that was eerily close to what subsequently transpired.

They then did nothing about it, and were caught completely off guard when their own scenario came to pass. 

When news of that fiasco came out, there was an entertaining game of pass the parcel in which each of the parties concerned tried to blame each other.

The British authorities had managed to snatch defeat from the jaws of victory!

Gillian Tett got it right when she described the tests as a predictable piece of public theatre.

Their purpose is to promote confidence – but whether that confidence is warranted is another matter. 

The stress tests remind me of those earlier pieces of predictable public theatre, the periodic suspensions of the Bank Charter Act of 1844: these worked as if by magic to calm the occasional panics of the period. 

Some of you might recall a delightful speech to the House of Commons by Benjamin Disraeli on August 30th 1848:
Just at the moment when unutterable gloom overspreads the population, when nothing but despair and consternation prevail, the Chancellor - I beg pardon, the Archbishop of Tarento - announces the liquefaction of the blood of St Januarius, … a wholesome state of the currency … return[s]: the people resume their gaiety and cheerfulness; …  everybody returns to music and macaroni … and in both cases the remedy is equally efficacious, and equally a hoax. 

However, the difference between the liquefaction miracle and the stress tests is that the liquefaction miracle is real.

3. The results of the 2015 stress tests

Lets start with the headline tests. 

Chart 1: Headline Results Based on the CET1 Ratio

This chart shows the test results based on the CET1 ratio using the Bank’s 4.5% pass standard. 

At first sight, everything looks well. 

However, the minimum pass standard should be at least as high as the regulatory minimum requirement, and this is 7%, not 4.5%.

Even the Bank’s own guidance document suggests that the pass standard in the stress test should be at least the minimum capital requirement!

As far as I can tell, every expert outside the Bank agrees on this point.

But still, the Bank insists that the pass standard should be 4.5% instead of 7%. 

To quote another of Lewis Carroll’s characters, when I use a word, it means just what I choose it to mean, neither more nor less. 

I would suggest this is scrambled logic. 

And his character was Humpty Dumpty – we all know what happened to him!

This is what we get with a 7% pass standard:

Chart 2: CET1 Ratio Stress Tests with a 7% Pass Standard


A very different picture!

Better still is to set the pass standard at the maximum possible regulatory standard under fully phased-in Basel III, taking account of the maximum possible countercyclical buffer of 2.5%.

The BIS has been warning us for some time about elevated risks: these suggest that a higher countercyclical buffer is called for.

Those concerned about elevated risks include the Chancellor. 

Anyone who thinks its mission accomplished with the UK economy is making a grave mistake, he said in January.

He then warned about a dangerous cocktail of new risks, and I agree. 

In any case, the pass standards need to be prudent to be convincing, and the more prudent, the more convincing.

Instead, the Bank gives the impression that it chose the lowest possible pass standard it thought it could get away with. 

This higher pass standard gives us the results in this next chart:

Chart 3: CET1 Ratio Stress Tests with Maximum Basel III Pass Standard


We now get quite a different picture! 4 banks fail, and only one clearly passes. 

However, these exercises are problematic because they depend on the RWA measure, and I think we can safely say that this measure is discredited – and not least by some brilliant work by the Bank’s chief economist, Andy Haldane. 

Chart 4: Haldane Chart on RWAs vs Leverage


As Andy’s chart here shows, RWA are actually contra-indicators of risk. 

Low RWA does not mean low risk, it means that such positions have risks that the risk measurement system cannot detect. 

There is a good reason for this too. 

Banks shift into low risk-weighted positions to get low capital requirements. 

They invent new securitisations that attract lower risk weights.

This game even has a name – risk-weight optimisation. 

You game the system to get the lowest possible capital requirements, and then siphon off the capital released as profits or bonuses. 

Then get a bailout when your bank goes bust.

So we need a denominator that is not RWA and this leaves us with a choice between total assets or leverage exposure. In practice, there is not much difference between these. 

This takes us to the second stress test used by the Bank – the leverage ratio test based on the ratio of Tier 1 capital to leverage exposure. 

This is what we get if we carry out a stress test using this leverage ratio measure:

Chart 5: Tier 1 Leverage Ratio Stress Test Results with a 3% Pass Standard


The average surplus over the pass standard is only 0.5%.

We also get two banks with surpluses of exactly zero – stress them any more and they fail. 

To make matters worse, these outcomes are biased because the numerator – Tier 1 capital – is unnecessarily soft. 

They are also biased because the denominator leaves out much of the off-the-balance risk.

Both of these create upward biases in the measured outcomes – this makes results look better than they really should be.

And there are big problems with the accounting numbers, which also make the banks seem stronger than they are.

You end up with false profits, illusory asset values and make-believe capital – and a set of IFRS compliant accounts may give no indication of the true state of your bank. 

We can’t do much about the denominator with available data, but we can harden the capital measure and use CET1 instead of Tier 1 in the numerator. This is what we then get:

Chart 6: CET1 Leverage Ratio Stress Test Results with a 3% Pass Standard

This gives us the next chart and the banks generally look weaker. 

The surplus falls to 0.2%. 

This is not a healthy specimen. 

Moreover, the 3% pass standard is itself extremely low. 

A 3% leverage ratio means that a loss of 3% is enough to wipe out all a bank’s capital. 

So the UK banking system looks distinctly vulnerable even by this weakest of leverage ratio standards. 

So what should the pass standard or minimum requirement be?

Again, it is important to apply a prudent standard, i.e., a high one. 

One answer is to apply the maximum possible standard under Basel III. 

This gives us an average pass standard of about 4.2%:

Chart 7: CET1 Leverage Ratio Stress Test Results with Maximum Basel III Pass Standard

By this standard, every bank fails the stress test. 

Another approach to the pass standard issue is to look at good practice overseas. 

For example, the Federal Reserve is due to implement a 5% minimum leverage ratio on the 8 biggest US bank holding companies in 2018. 

This is not a particularly high standard either: in the US, 5% is the minimum needed for a bank to be considered ‘well-capitalised’ under the Prompt Corrective Action framework.

Yet even single UK bank would fail this stress test.

The new Fed requirements will also impose a 6% minimum on the federally insured subsidiaries of those big banks.  

But I would go further and suggest that even these pass standards are way too low. 

On this subject there is a remarkable degree of consensus among experts. 

For once, economists agree!

To give an example, in 2010 there was a letter in the FT signed by 20 world leaders in the field. 

This letter advocated a minimum leverage ratio of no less than 15%, five times the pass standard applied by the Bank of England. 

And many of the signatories on other occasions suggested much higher minima still. 

And I can think of other experts who would agree with them. 

4. An alternative to stress testing

So how best to gauge banks’ capital adequacy?

The answer is old-fashioned capital ratio analysis. 

Compare actual robustly conservatively estimated ratios against reasonable pass standards like, e.g., 15%. 

No stress test, no stress! Just actual data. 

We then get an average leverage ratio across the banking system of 4.3%:

Chart 8: 2015Q3 CET1 Leverage Ratios Against a 3% Pass Standard

And the average leverage ratio falls to under 3.7% if we use the market value instead of the book value.

A simple capital ratio analysis using latest data then reveals the weakness of the banking system that the Bank of England stress tests manage to completely miss.

There are many other points to be made but I should probably bring this to a close. 

5. Take-home

So here are the take-home points. 

First, RWA is meaningless – and so are its progeny.

You should never believe anything based on RWA metrics – this includes the RWA-based stress tests.

Away with the pixies basically… 

Second, the stress tests lack credibility because they are based on very low pass standards – and the entire system fails the test if we use more reasonable pass standards instead. 

Third, we don’t need any stress tests at all. 

Stress tests merely provide false risk comfort and distract everyone from the true state of the banking system and the dangers we face. 

Fourth, banks are still massively undercapitalised

So my advice to the Bank of England:

Stop the stress tests, now.

Raise minimum required leverage ratios and stop banks making any distributions of dividends or bonuses until they meet those requirements.

And my advice to policymakers generally:

Get the numbers right – we need reliable accounting standards

Lastly, get the incentives right – we can only ever expect bankers to behave themselves if they face real sanctions when they blow up their banks.

Thank you all. 

Kevin Dowd
March 17 2016