Oil price forecasting by EIA and Merrill Lynch not going so well

Forecasting is a tricky business. There are many ways to do it wrong due to the many biases that affect the forecaster. Oil forecasting isn’t just a statistical problem solved with numbers, there has to be some domain expert knowledge applied and applied carefully.

Once a forecast is made, time will judge its accuracy. Use MAPE (Mean Average Percentage Error) as the yard stick, we can quantify the accuracy of past forecasts. Almost all forecasts will be off my some amount, but we really want to find a forecast method that gives the smallest error over time. Hence, we seek one with the smallest MAPE.

Below is a graph of the error of the EIA (Energy Information Agency), Bank of America’s Merrill Lynch, and a simplistic model called Naive.

The EIA and BoAML can't even beat the Naive model for forecasting 1Q, 2Q, 3Q, and one year out.

The EIA and BoAML can’t even beat the Naive model for forecasting 1Q, 2Q, 3Q, and one year out.

The Naive model simply looks at the price of oil the day before the forecast and says that is what it will be 3 months from now, 6 months from now, etc. There are 4 simplistic models in forecasting theory that set the bar low. To illustrate how simple they are, two of them are the mean price model, and the naive model. If your complex proprietary model can’t even beat (achieve lower MAPE) the simple four, you should give up.

What this graph is saying, is that a sophomore in high school could have made a better oil forecast than the EIA or Merrill Lynch. Except forecasting 18 months out, in which case Merrill Lynch redeems themselves.

In another post we’ll review EIA’s performance over a longer period of time than just 5 years.