Forecasting – The art of being precisely inaccurate

My column in the Economic times on the 30th of sep

The unedited version is here :

Of late, equity markets have behaved as someone with a multiple personality disorder – euphorically up one day, manically depressed the next. Much of this movement is attributed to market moving policy pronouncements. Equally, volatility arises when key economic numbers are released.

Talking heads on TV often attribute market moves to differences between forecast economic numbers (in particular, GDP growth and inflation) and the “actual”. But are the forecast figures really useful for market participants?

What makes a good forecast?

An obvious requirement for a good forecast is its “accuracy” - how good was the forecast compared to the actual outcome? Since the “event” is yet to occur, another important consideration is if the forecast is “honest”. In other words, was it the best prediction the forecaster could make when making the forecast, or was it deliberately coloured or biased. A third factor could be the “value” of the forecast – did it convey information that was useful in making decisions.

Understanding predictions

When dealing with the future, we deal with probabilities. In making a forecast, the forecaster deals with multiple scenarios, and, either explicitly or implicitly, assigns a probability to each scenario. A “point forecast” –a weighted average of future expectations – is, usually, less useful.  An old joke goes – a statistician drowned while crossing a river that was three feet deep on average! Making dramatic statements with high degree of certainty makes for good television, and studies show that “bold” predictions make for better entertainment. They certainly do not guarantee greater accuracy though, or help in better decision making.

A wide distribution of possible outcomes best represents the uncertainty inherent in the real world. Our minds however tend to regard probability based forecasts as somehow not so satisfying –as if the forecaster is “hedging” his bets.

Professional Forecasting – dismal performance

Starting 2007, the Reserve Bank of India conducts a survey of professional economists. Those surveyed offer forecasts for a number of macro-economic variables. These forecasts are made every quarter, and the RBI publishes the results on its website. Among other variables, the professional forecasters offer their estimates for GDP growth, its key components, and inflation.

The forecasts for GDP are made in the form of a probability table. RBI combines the forecasts to yield a min and max forecast range, as well as the median. The picture summarises GDP growth forecasts made in April-May for the next 12 months for every financial year since the survey started.

A quick glance reveals that forecasters do not cover themselves in glory. In each of the 5 years where data is available, the actual GDP growth is outside the maximum-minimum range forecast. Take a minute to digest that – it is not just different from the median, it is outside the forecast range – and in many cases by a wide margin.

Governments own forecasters – even worse?

The prime ministers economic advisory council (PMEAC) too, makes estimates for the same figures twice a year. Unfortunately, it offers a “point estimate” instead of a range. While this gives the perception of being more decisive and accurate, as the table shows, their track record is equally poor – with some estimates having an error of over 10% from the actual - when made only 30 days before the end of the financial year!

GDP Growth
Date Of PMEAC  Report				Forecast for	Actual
Outlook		Review
30-Jul-08	7.7	Jan-09	7.1	Mar-09	6.7
Oct-09	6.5	Feb-10	7.2	Mar-10	8.6
Jul-10	8.5	Feb-11	8.6	Mar-11	9.3
Jul-11	8.2	Feb-12	7.1	Mar-12	6.2
Aug-12	6.7	Apr-13	5	Mar-13	5
Aug-13	5.3			Mar-14

The one redeeming feature is that forecast directions seem to be correct – each subsequent forecast moving in the direction of the eventual number.

Important forecasts, but with spurious accuracy

Economic forecasting is important in that the forecast can itself affect the outcome as policies are adjusted to move the economy in the desired direction. What causes these forecasts to be as poor as they are?

Assuming that forecasters are not wholly incompetent, forecast errors can arise out of three possible factors–

(a) data used to make forecasts is of poor quality. Remember, revisions to the final GDP growth continue for almost 2 years – and advance estimates are notoriously poor. This calls in question the importance attached to near term data. When RBI/government say that they will formulate policy basis data, it is only fair to ask – what data?

(b) forecasts are biased. This is also a real possibility. Median forecasts tend to cluster around the 6%-8% growth. Forecasts are understated in years of high growth and overestimated in those of poor growth. There is a visible tendency to cluster around the “centre”. It seems no one wants to rock the boat. But then, this “conservatism” also reduces the value of forecast in policy formulation.

(c) models are poor – an economy is a dynamic system feeding on itself and other external stimuli. Models attempting to replicate economic performance have to factor in a large number of inputs. Additionally, models are likely to be susceptible to “initial conditions” – which themselves may be sources of errors.

In all cases, the important point that the investor needs to note is that market volatility caused by data releases is largely unwarranted. It provides the patient investor an opportunity to make abnormal gains by betting against short term moves. It is important to remember that the apparent accuracy of most economic numbers is a mirage – and the best one can use them for is to determine the direction of the trend.