Forecasting – The art of being precisely inaccurate

My column in the Economic times on the 30th of sep

The unedited version is here :

Of late, equity markets have behaved as someone with a multiple personality disorder – euphorically up one day, manically depressed the next. Much of this movement is attributed to market moving policy pronouncements. Equally, volatility arises when key economic numbers are released.

Talking heads on TV often attribute market moves to differences between forecast economic numbers (in particular, GDP growth and inflation) and the “actual”. But are the forecast figures really useful for market participants?

What makes a good forecast?

An obvious requirement for a good forecast is its “accuracy” - how good was the forecast compared to the actual outcome? Since the “event” is yet to occur, another important consideration is if the forecast is “honest”. In other words, was it the best prediction the forecaster could make when making the forecast, or was it deliberately coloured or biased. A third factor could be the “value” of the forecast – did it convey information that was useful in making decisions.

Understanding predictions

When dealing with the future, we deal with probabilities. In making a forecast, the forecaster deals with multiple scenarios, and, either explicitly or implicitly, assigns a probability to each scenario. A “point forecast” –a weighted average of future expectations – is, usually, less useful.  An old joke goes – a statistician drowned while crossing a river that was three feet deep on average! Making dramatic statements with high degree of certainty makes for good television, and studies show that “bold” predictions make for better entertainment. They certainly do not guarantee greater accuracy though, or help in better decision making.

A wide distribution of possible outcomes best represents the uncertainty inherent in the real world. Our minds however tend to regard probability based forecasts as somehow not so satisfying –as if the forecaster is “hedging” his bets.

Professional Forecasting – dismal performance

Starting 2007, the Reserve Bank of India conducts a survey of professional economists. Those surveyed offer forecasts for a number of macro-economic variables. These forecasts are made every quarter, and the RBI publishes the results on its website. Among other variables, the professional forecasters offer their estimates for GDP growth, its key components, and inflation.

The forecasts for GDP are made in the form of a probability table. RBI combines the forecasts to yield a min and max forecast range, as well as the median. The picture summarises GDP growth forecasts made in April-May for the next 12 months for every financial year since the survey started.

A quick glance reveals that forecasters do not cover themselves in glory. In each of the 5 years where data is available, the actual GDP growth is outside the maximum-minimum range forecast. Take a minute to digest that – it is not just different from the median, it is outside the forecast range – and in many cases by a wide margin.

Governments own forecasters – even worse?

The prime ministers economic advisory council (PMEAC) too, makes estimates for the same figures twice a year. Unfortunately, it offers a “point estimate” instead of a range. While this gives the perception of being more decisive and accurate, as the table shows, their track record is equally poor – with some estimates having an error of over 10% from the actual - when made only 30 days before the end of the financial year!

GDP Growth
Date Of PMEAC  Report				Forecast for	Actual
Outlook		Review
30-Jul-08	7.7	Jan-09	7.1	Mar-09	6.7
Oct-09	6.5	Feb-10	7.2	Mar-10	8.6
Jul-10	8.5	Feb-11	8.6	Mar-11	9.3
Jul-11	8.2	Feb-12	7.1	Mar-12	6.2
Aug-12	6.7	Apr-13	5	Mar-13	5
Aug-13	5.3			Mar-14

The one redeeming feature is that forecast directions seem to be correct – each subsequent forecast moving in the direction of the eventual number.

Important forecasts, but with spurious accuracy

Economic forecasting is important in that the forecast can itself affect the outcome as policies are adjusted to move the economy in the desired direction. What causes these forecasts to be as poor as they are?

Assuming that forecasters are not wholly incompetent, forecast errors can arise out of three possible factors–

(a) data used to make forecasts is of poor quality. Remember, revisions to the final GDP growth continue for almost 2 years – and advance estimates are notoriously poor. This calls in question the importance attached to near term data. When RBI/government say that they will formulate policy basis data, it is only fair to ask – what data?

(b) forecasts are biased. This is also a real possibility. Median forecasts tend to cluster around the 6%-8% growth. Forecasts are understated in years of high growth and overestimated in those of poor growth. There is a visible tendency to cluster around the “centre”. It seems no one wants to rock the boat. But then, this “conservatism” also reduces the value of forecast in policy formulation.

(c) models are poor – an economy is a dynamic system feeding on itself and other external stimuli. Models attempting to replicate economic performance have to factor in a large number of inputs. Additionally, models are likely to be susceptible to “initial conditions” – which themselves may be sources of errors.

In all cases, the important point that the investor needs to note is that market volatility caused by data releases is largely unwarranted. It provides the patient investor an opportunity to make abnormal gains by betting against short term moves. It is important to remember that the apparent accuracy of most economic numbers is a mirage – and the best one can use them for is to determine the direction of the trend.

The dragon kings of Didier Sornette and the skill of predicting financial bubble bursts

This article was carried in the economic times on 2nd august

Chances are you have never heard of Didier Sornette. Chances are that the next time you hear of him, you will be significantly poorer – unless you listen to what he says now!

Sornette is Professor of Entrepreneurial Risks at the Swiss Federal Institute of Technology (ETH Zurich). His research focuses of the prediction of crisis and extreme events in complex systems. His work covers earthquake physics, dynamics of success on social networks, and complex system approach to medicine. However, the part that attracts market attention is what he does at the Financial Crisis Observatory – which is – to test the hypothesis that financial “bubbles” can be diagnosed in real time and their termination predicted probabilistically. In other words, he attempts to find when the next big fall in the financial markets can occur.

The term “bubble” refers to a situation where excessive future expectations lead to rise in prices. Sornette identifies speculative bubbles as arising from a confluence of two factors – factors that drive initial demand – say, new technology, or perception of reduced market risk. This is followed by “amplification mechanisms”, where large increase in asset price is followed by higher demand as investor think that further large increases in price will follow. This “super-exponential” acceleration in prices due to a positive feedback (or “pro-cyclicality”) leads to formation and then maturation of a bubble in finite time.

In other words when expectations of growth rate itself grow, it leads to instability. Recent examples have been the crash of 2008, and the technology burst in 2000, among others. In Sornette’s world, the cause of the crash is unimportant. His research suggests that crashes have an internal origin – the unrealistic rise in expectations – and external factors only serve as catalyst for the subsequent burst. So why is all this important?

Of Black Swans...

In 2001, Nassim Nicholas Taleb, quantitative trader and academician published a book “Fooled by Randomness” where he outlined the theory of “Black Swans”. Taleb described “Black Swans” as events whose probability of occurrence was mathematically very low (like finding a black swan in a bevy of white swans). Taleb explained that these events occur with higher frequency than theory predicted, were hard-to-predict if not impossible to predict, and caused events of significant consequence and magnitude.

... and Dragon Kings

Sornette, on the other hand, makes an entirely contrary claim. Not only can he predict the probability of a bubble bursting, he can do it with remarkable accuracy and of course, before the event! He calls these “outlier events” as Dragon Kings.  The graphs below show the predictions of the S&P500 US Index, and oil prices – made before the crash in 2007-2008. Similar other graphs can be found on the website of his Financial Crisis Observatory.

 

A matter of modelling

The broad basis of the prediction is based on “power law”. Most models of market prices use the “normal” distribution to model price behaviour. This model underestimates risk. Studies suggest that a better model, especially when markets are leveraged – which they often are – is to apply the power law.

 

 

Sornette’s model looks for “out-of-control” growth in asset price that vary from the power law. “When herding behaviour among investor’s ramps up, a stock’s or index’s growth rate can increase faster than exponentially, leading to more herding. This positive feedback brings the system to a tipping point. About two-thirds of the time, a crash results”, says Sornette in a paper in 2009.

To break away from allegations that his forecasts are self-fulfilling – after all, market participants who believe his forecasts are likely to start positioning themselves accordingly, Sornette’s team now makes forecast which are released in encrypted form on a website with a public key to decode the paper after the forecast period. . His most recent prediction was a “”Sell” signal on 21 May 2013 on the S&P500, when his Crash Risk Index jumped up (see graph). The market was down 9.5% in a month after that.

Sornette recently made a presentation at TEDGlobal 2013 – a talk worth viewing (http://www.ted.com/talks/didier_sornette_how_we_can_predict_the_next_financial_crisis.html). The upshot of the presentation and the subsequent interview (http://blog.ted.com/2013/06/17/turbulent-times-ahead-qa-with-economist-didier-sornette/)

is that he continues to foresee bubbles in financial and insurance sectors, as well as construction and realty sectors in the USA – the very same sectors that led the burst in 2008 in the first place. Ironically, the success of a model can also lead to its demise as participants adjust their behaviour to include the forecasts of the model. Till this happens, Sornette’s research needs to be taken seriously.