How Do You Know It Works (part 3)?

In this 3rd part of  ‘How Do You Know It Works” I’m going to cover three different popular ways to measure risks and show how to tell if they work or not (I know, I know, that part 2 cartoon is a cheap way of making this a trilogy-post but I’ll take what I can get).

What we’re talking about is much wider than financial risk, but it’s also at the very heard of financial risk management:  How do you know if your risk measure is worth the paper it’s printed on or the computer it runs on?  Here are some things to look for in evaluating a particular forward-looking risk analytic:

1. It can be calculated from available information and isn’t tied to a proprietary piece of data.
2. The analytics’s accuracy can be quantitatively determined in some way.
3. The assumptions going into the analytic can be clearly communicated.
4. Users of the risk analytic can have an indication of when it is applicable and when it is not.

I’m just going to concentrate on the second feature: evaluating the quantitative accuracy of the analytic.  Let’s consider three different analytics:

Worst Case Loss
Investor Analytics is sometimes asked to calculate a firm’s proprietary risk model which falls into the category of “worst case loss.”  I’ve heard it described many different ways, using many different formulas / techniques / methodologies, and it usually starts with a set of reasonable inputs but boils down to a series of alchemy-like adjustments that render the thing useless.

Here’s an example: “take every security in my portfolio and get their worst day over the last 2 years, or 3 years or 5 years [little voice in my head: so far so good – that’s both reasonable and can be calculated].  Then, take the worst performance that those securities ever had over those years and pretend they all happened on the same day [wait a minute – we’re mixing apples and oranges…].  Then, add up all those worst day losses and multiply by 3 [three? why three?!?  why not 5, or π, or any other number???].  That’s what we mean by the portfolio’s worst case loss.  Can you do that for us?”

At this point in the conversation, I usually have to control myself so I don’t end up offending the person.  So I start asking some questions, and the dialogue goes something like this:

Me: “So, how many times in the past few years has your portfolio lost that amount?”
Them: “Oh, never.  That would be catastrophic.”

Me: “OK.  How many times has it come close to losing that amount?”
Them: “Not once.”

Me: “OK.  I’m not sure I fully understand – hopefully you can help me.  If your portfolio has never actually lost that amount, how do you know that the number is correct?  I mean, how do you know that 2 times that number isn’t better?  Or 1/2 the number?”
Them: “Well, we’ve been using this measure and it’s worked for us.” [Ah, the kiss of death: they are emotionally wedded to this method].

Me: “OK.  I understand.  Are there any other ways you measures the risk in your portfolio?”
Them: “No, this is it.”

This is a really bad situation.  They are convinced that the method works because their portfolio has never lost that amount.  It reminds me of that episode of Gilligan’s Island where Gilligan finds some talisman that wards off vampires [or tornadoes or whatever it was supposed to ward off].  When the Professor (dare I say he was my second favorite character – I always had a thing for Mary Ann) challenged him on this, Gilligan said something like “see – it works!  This island has no vampires [tornadoes, whatever].”

This “worst case method” is designed so it can’t be tested. If it can’t be tested, how can you ever know if it’s true or not?  Any so-called ‘measure’ whose accuracy cannot be tested is basically worthless.

Value-at-Risk
I’m only going to consider how you test Value-at-Risk, and I won’t get into all the arguments pro- (yes, there are some) and against- VaR.  Quick review: a portfolio’s Value-at-Risk is an estimate of how much money the portfolio will lose on a really bad day.  How bad of a day?  That’s a key point: let’s say the 95% (1-day) VaR is $1,000,000.  That means that over the next day, your portfolio should make money (yeah!) or, if it loses money, it shouldn’t lose more than $1M.  Oh, and when I say “shouldn’t”, I mean that it won’t lose more than that amount 95% of the time.  In other words, 5% of the time, you should lose more than $1M.  Did you catch that?  If something doesn’t happen 95% of the time, then it does happen the other 5% of the time.  That fact right there provides a way to test the VaR number: does your portfolio actually lose more than that number 5% of the time?  5% is 1 day in 20, or 1 trading day per month.  So, every month, your portfolio’s loss should exceed the VaR number once.  These are averages, so it doesn’t have to be each and every month.  But it does have to be close to 1 day a month, or about 5 days out of any consecutive 100 days.  Not 8 days.  Not 2 days.  Right around 5 days.  For the 99% VaR number, it’s one day in one hundred, or about 3 1/2 times a year.

[Side note: for the 95% VaR, some people say that the portfolio’s losses shouldn’t exceed the VaR amount more than 5% of the time, but that it’s OK as long as you exceed it less than 5% of the time.  They think it’s perfectly fine to go 3 or 4 or 5 months without a loss bigger than the 95% VaR number.  That’s totally wrong.  If you go, on average, 4 months without losing the VaR number, then that means you’ll exceed the VaR number about 3 times a year.  Well, that means your number behaves like a 99% VaR, and NOT like a 95% VaR.  That’s a very important distinction.]

But the point of this is example is that VaR can be tested, which means that certain ways of calculating VaR can be shown to work, while others can be shown not to work.  The European Regulators who came up with UCITS understand this very well and they even incorporated back-testing of VaR into the regulation.  That makes sense.  Everybody should backtest VaR.  It’s easy and there’s no excuse not to.

Historical Scenario Analysis
This type of risk analysis is rather interesting because many people say they like it, many professionals want it in their risk toolbox, there are in fact ways to test it but nobody really ever bothers to do so.  Basically, historical scenarios are an attempt to replay historical crashes and determine how the current portfolio would fare if history were to repeat itself.  Of course, there’s no way to recreate every single aspect of the historical situation, but there are ways to simulate quite a few of the more important ones.  So people come up with a list of their “top 10” historical scenarios and this analysis gives them the comfort of estimating what would happen if those events repeat.

How do you test something like this?  Since you can use the technique to estimate the portfolio’s returns over a specific time range, then you can cleverly choose a very recent time range and keep tabs on how well the simulation does as those time periods happen.  This is basically using a back-testing approach.

I’m not a terribly big fan of historical scenarios, mainly because history will not repeat itself.  It seems to me that a better way of doing this sort of analysis is to examine interesting crash periods and identify what were the drivers of the crash.  Taking those drivers, examining how they related to each other before the crash, during the crash and after the crash (in price, volatility, correlation, etc.) will allow you to construct nightmare scenarios that are much more realistic than thinking that the particular crash will repeat itself.

But people are largely comforted by knowing that if the same boogieman who came on Oct 19, 1987 comes again that the portfolio will be safe.

Closing Point
Testing – counting the times you’re right and the times you’re wrong – is the only way that we humans have developed that consistently improves things.  As I wrote in part 1 of this post, if something fails a well designed test, then the only correct thing to do is to: stop using it, stop believing it, stop thinking it’s right and stop making decisions based on it.  This is harder than you think, especially if you yourself came up with the method that just failed.

But the first step is testing.  Without testing your risk analytics there is no way of knowing if you should follow their warnings or not.  And in that case, what’s the point?