The Six Fundamentals of Accurate Forecasting
by Philip Stubbs, Drakelow Consulting
In this article I’m presenting what I consider to be the six fundamentals of accurate and useful forecasting. These are the essential elements that must be in place in order to achieve excellent forecasting results. While reading this article, I ask you to study these six fundamentals, and critique your own organisation’s capability within each one.
But first… a bit about my own background. I studied forecasting at a postgraduate level at Lancaster University Management School. This was a very useful grounding in many important forecasting principles. As part of my studies, I conducted research into the short-term forecasting of gas demand for British Gas, where I tried out many different approaches until I found the most accurate method. The project improved the forecasting accuracy significantly, and the research was published in a world-leading academic journal.*
Since then, I have spent over 25 years using many different forecasting techniques and approaches to solve a wide range of different problems in industries such as retail, telecoms, utilities, and finance. This article provides a very brief outline of what I have learnt along the way!
The first important fundamental for accurate forecasting is up-to-date, reliable data. It is far better to forecast by applying a simple model to accurate data, rather than to apply a sophisticated model to unreliable data. Using poor data will always result in larger errors.
The first dataset to study is the history of the forecast variable, ie what you are trying to forecast. It is essential to check the integrity of this data regularly, to make sure that this data is what you believe it to be. Also, make sure that you have the most up-to-date information, since if you make a forecast without a recent data point, your forecast may not take into account a movement or change that has just happened.
Ideally, all of the variation in the forecast variable data series should be explainable as seasonality, trend or some external factor. I would recommend keeping a diary of all variation, because the reasons can quickly get forgotten in time. The key to accurate forecasting is understanding the past.
Where the historical variation might be explained by one or more independent variables, or drivers, then you need up-to-date and accurate data of these as well.
Number two in our list of forecasting fundamentals is the forecasting model. This is the equation or algorithm you use to turn the data and business insight into the most accurate forecasts possible.
There are two general modelling approaches to forecasting: the first to consider is Time Series, where you only use the historical values of what you are trying to forecast. So if you are forecasting future emails, you would only use the historical data of email volumes.
Time Series models allow you to smooth historical data, and also deal with trend and seasonality. Such seasonality may be within-day, or it may be regular variation you see across a year. There are many different types of Time Series model to use here, from simple moving averages to exponential smoothing, and then onto more complicated techniques such as single-variable ARIMA models.
A second forecasting approach is Explanative modelling, which is useful where variation in volume is caused by one or more other factors. For example, when inbound sales calls are caused by marketing campaigns, or when changes to Customer Service call volume are provoked by variation in statement despatch. Examples of Explanative models are simple ratio models, propensity modelling, regression-based models and machine learning.
Very often, I have found that organisations use very ill-suited Explanative models that can be outperformed by simple Time Series models. I will explain the reasons how and why this happens in another article, but often the poor model results from a forecast being made with reference to a business indicator when there is no good relationship between the forecast variable and the indicator.
You must use robust methodology to select the optimum model. This will involve building a number of candidate models using a historical dataset. Then, test each of these models with a holdout dataset, where this holdout data must not have been used to build the models. It is essential to test the candidate models on such holdout data. This is because often the most accurate model is not the one that best fits the historical data.
The optimum model for forecasting gas demand was based on forecast values of temperature and windspeed from the Met Office. Also, the day of the week was a significant factor too. I also introduced a feedback mechanism where a proportion of the previous day’s error was fed back in to the forecast, since I identified a pattern known as autocorrelation in the model’s errors. So for gas demand, the optimum approach was a mixture of both Time Series and Explanative approaches.
In most forecasting situations, there is a requirement for regular forecasts, rather than a one-off piece of analysis. This is because forecasts are needed to make decisions, and the decisions may need to be made regularly. For example, the forecast demand for a stocked item may be needed weekly, if there is a weekly calculation for whether more stock needs to be purchased.
Where there is a need for a regular forecast, a robust process is required to ensure that good quality input information is delivered on time, that the steps are carried out correctly, and that the outputs are reviewed and used properly. Here are four examples of process elements that improve forecast accuracy:
The first consideration is that the forecast analyst should negotiate with the customer to determine the latest possible time that the forecast is needed. This means that the regular forecast can be created taking into account the largest amount of recent data as possible.
Second, a process is required with all information suppliers to ensure any driver forecasts are available on time, and are critiqued and challenged appropriately, perhaps in a regular meeting.
Third, when the forecast is ready, it should be critiqued and challenged by stakeholders. There is often a piece of relevant information only given by a stakeholder at the point of reviewing a forecast. If it is material, then the forecast should be revised. The application of judgement has often been shown to make forecasts more accurate, but this must be evidence-based and well-controlled.
Finally, an error monitoring process is vital, where recent historical forecasts and actual values are compared, so that forecast variances are learned from – a feedback loop to enable all insight to be gained from the variances. This can be done with support from stakeholders.
Each of these process considerations exist to ensure the relevant communication takes place to maximise forecast accuracy.
There are many tools available today that offer forecasting automation, for example the forecasting components of customer service Workforce Management tools. Such tools are convenient to use because they take data directly from workflow systems, and then make forecasts automatically within the planning system. Too often, however, the range of forecasting models on offer is restricted to one or just a few Time Series models. If you prize accuracy over convenience, then it's likely that it will be necessary to perform the forecasting outside of the WFM, and overwrite the WFM’s own forecasts.
Excel includes many features to perform forecasting, and is commonly used for that purpose. It can hold historical data and a wide range of Time Series and Explanative forecasting models can be built using Excel’s formulas and functions. Further, Excel 2016 introduced a forecasting tool that applies a method similar to a Time Series model called Holt-Winters, which calculates forecasts by applying historical trend and seasonality into the future. There is also a multiple regression tool in Excel, that allow you to build forecasting models with one or more input variables, and also test the statistical significance of those input variables. The downside of Excel includes the problem of transferring analysis results into other systems, but analysts will find the best way to automate the transfer as much as possible.
For those situations where more complex Time Series and Explanative models are required, I have always found SAS to be quick, powerful and full of functionality and insight. It quickly supplies optimum parameters for advanced Time Series techniques, and it can guide you through the more exotic Explanative models. There are other tools available, including R which has the advantage of being completely free.
Cheap and effective, the use of charts can offer a huge help to maximising accuracy. I cannot overstate the clarity and insight they bring to help with learning, understanding, checking and building consensus. When I have visited organisations that have not used charts for forecasting, their introduction has led to improvements in accuracy. Here are just two areas of many where they can help:
- Understanding Variation – creating a chart of historical data of the forecast variable helps you to build an awareness of the causes of all historical variation, which is essential to accurate forecasting
- Reviewing and checking forecasts - sharing forecasts with stakeholders, together with recent actuals on a chart can help spot improvements, build consensus, and bring out new suggestions
I’m often mocked for preaching about the use of charts to improve forecasting, but I have seen so frequently how the implementation of the appropriate chart can make forecasts more accurate. Never forget the power of the chart.
The analyst responsible for forecasting must have good knowledge of different forecasting approaches, and also be aware of best practice to select the optimum model. Advanced knowledge of Excel is essential and also, ideally, experience of the statistical software packages. With a skilled and knowledgeable leader, coach or trainer, forecasting skills can be developed quickly by a capable analyst. Competencies required by a forecasting analyst include Attention to Detail, Communication, Influencing, and Planning & Organising.
Further, the forecasting stakeholders should give support to the analysts by providing insight, assisting the process where required and reviewing the forecasts.
Making sure the right skills and behaviours are in place, within both the analyst and stakeholder communities, are essential to making forecasts useful and accurate.
Have you taken a view of the capability of these six fundamentals in your organisation? If you would like to discuss how to improve the accuracy and usefulness of your forecasting, across any or all of these fundamentals, then please contact me at firstname.lastname@example.org
* One day ahead demand forecasting in the utility industries, Journal of the Operational Research Society 48: 15-24 (1997)