How Do You Know It Works?

I participated in a panel yesterday at a hedge fund conference in NYC.  What made this panel a little different was that I wasn’t the first person to talk about how important it is to know the limitations of risk models.  In fact, I was the THIRD person.  The question was posed “what makes an outstanding risk manager?”  The first person to respond included in his answer something along the lines of “an outstanding risk manager asks the question – how do I know the model works?”  Bingo!

How do you know if any prediction works?  Over the ages, people have tried all sorts of methods.  Let’s take a look a few of them.

Method #1: you believe it if an authority figure tells you it’s true. Like the village elder.  Or the shaman.  Or a celebrity.  Or Congress.  Assessment: inconsistent results at best.  Let’s try something else…

Method #2: you believe it if everybody else (you know) believes it. Like for thousands of years that the earth is flat.  Or that the sun goes around the earth.  Or that a big bearded guy goes around the world one night a year dropping off exactly what you asked for.  Assessment: this method gives ‘comfort in large numbers,’ but is basically worthless.  Let’s try something else…

Method #3: you believe it if it seems reasonable. Like that a portfolio manager named Bernie can make 1% returns every month without ever losing a penny.  Criticizing this method is a bit tricky because sometimes it works, but there are many things that seem reasonable that end up not being true.  Assessment: this method isn’t all bad but it certainly isn’t a good way of determining if something works.  Let’s look for something else…

Method #4: you believe it if you have some evidence for it. Like if one vaccinated child (out of tens of thousands) later shows signs of autism, then clearly the disease must be caused by the vaccination.  Lots of things have some evidence in their favor, and this method is on the right track but it doesn’t get us far enough.  Why?  Because people – all of us – are easily convinced of something if there’s a little evidence for it.  What we often fail to do is examine the evidence against the thing.  Take astrology (please!) – there is some evidence for it, it seems.  But on closer inspection, there is far more evidence against it, and it turns out to be no better than random.  Assessment: method #4 is useful as a negative filter – you can discard things that have no evidence, but just because there’s some evidence doesn’t mean the thing is true.  So let’s move on…

Method #5: You test it. Wait a minute… this one actually works!

Of course you test it.  If you have a way of predicting something, you set up a test to measure how often you’re right against how often you’re wrong.  You have to count the times you’re wrong. This is done for everything from medical tests to engineering projects to election polls to new inventions.  Let’s take a look at an example in detail: a scary blood test result.

Image your doctor tells you that a recent blood test came back indicating that you have a serious disease, and you need to be checked into the hospital to begin treatment.  You may decide, quite reasonably, to check yourself in that day.  Now image, instead, that your doctor tells you that a recent blood test came back indicating you have a serious disease but that the blood test has a 20% false-positive rate.  In this case, your decision is probably very different – you’ll probably want another test before deciding to check into the hospital.  Knowledge of that false-positive rate really matters.  It’s a measure of the test’s accuracy, and it HAS to be quoted to you for you to make an informed decision.  Notice that the test results are the same in both cases: both tests indicate you have a serious illness.  In the first case, it seems all but certain.  In the second case, there’s a quantified assessment of how likely that result is to be true.  Now that’s something useful!  To get that false-positive rate, people had to compare this blood test against other ways of determining if patients actually have the illness, and with enough careful testing you can establish a quantified assessment of a particular prediction’s accuracy.  In fact, this is the only way to do it correctly.

In risk management, we’re often concerned with predicting how much money a financial portfolio can lose.  Regardless of which measure is actually used, it is basically making a prediction.  So the right thing to do is to test it, and test it regularly.  The test is basically done by comparing the risk measure’s prediction with what actually happens in the future: by counting the “hits” and the “misses.”  Done this way, the prediction’s accuracy can be measured quite well.


METHOD #5A: You discard tools/beliefs that fail the test. This may seem obvious, but it is very hard for humans to do.  What do I mean?  Once we’ve attached ourselves to believing that something works, it’s very hard to let go, even in the face of contradictory information.  We tend to keep the status quo.  For us, change is hard.  But once we know that a method doesn’t work, no matter how much we are attached to that method, the only behavior that actually improves things is to stop using the method.

What if there is no alternative to a method that fails the tests?  Isn’t doing something better than doing nothing?  My answer is a very clear NO.  If you lived in the 1700’s you might be tempted to bleed a loved one who’s ill.  “Well, it may not work all the time, but it’s better than doing nothing.”  Not really.  Today, using risk tools that fail a test is tantamount to bleeding a person. It puts a false sense of security around something that may actually cause more damage.  My next post will go over a few different examples of testing three common risk measures: a “worst case” loss, a “Value-at-Risk” number and “Historic Scenario Recreation.”

One Response to How Do You Know It Works?

  1. Pingback: How Do You Know It Works (part 3)? « Risk-ology

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: