Forecasting is a tricky business. There are many ways to do it wrong due to the many biases that affect the forecaster. Oil forecasting isn’t just a statistical problem solved with numbers, there has to be some domain expert knowledge applied and applied carefully.
Once a forecast is made, time will judge its accuracy. Use MAPE (Mean Average Percentage Error) as the yard stick, we can quantify the accuracy of past forecasts. Almost all forecasts will be off my some amount, but we really want to find a forecast method that gives the smallest error over time. Hence, we seek one with the smallest MAPE.
Below is a graph of the error of the EIA (Energy Information Agency), Bank of America’s Merrill Lynch, and a simplistic model called Naive.
The Naive model simply looks at the price of oil the day before the forecast and says that is what it will be 3 months from now, 6 months from now, etc. There are 4 simplistic models in forecasting theory that set the bar low. To illustrate how simple they are, two of them are the mean price model, and the naive model. If your complex proprietary model can’t even beat (achieve lower MAPE) the simple four, you should give up.
What this graph is saying, is that a sophomore in high school could have made a better oil forecast than the EIA or Merrill Lynch. Except forecasting 18 months out, in which case Merrill Lynch redeems themselves.
In another post we’ll review EIA’s performance over a longer period of time than just 5 years.