Exponential smoothing

Exponential smoothing or exponential moving average (EMA) is a rule of thumb technique for smoothing time series data using the exponential window function. Whereas in the simple moving average the past observations are weighted equally, exponential functions are used to assign exponentially decreasing weights over time. It is an easily learned and easily applied procedure for making some determination based on prior assumptions by the user, such as seasonality. Exponential smoothing is often used for analysis of time-series data.

Contents

• Basic (simple) exponential smoothing
• Time constant
• Choosing the initial smoothed value
• Optimization
• "Exponential" naming
• Comparison with moving average
• Double exponential smoothing (Holt linear)
• Triple exponential smoothing (Holt Winters)
• Implementations in statistics packages
• See also
• Notes
• External links

Exponential smoothing is one of many window functions commonly applied to smooth data in signal processing, acting as low-pass filters to remove high-frequency noise. This method is preceded by Poisson's use of recursive exponential window functions in convolutions from the 19th century, as well as Kolmogorov and Zurbenko's use of recursive moving averages from their studies of turbulence in the 1940s.

The raw data sequence is often represented by ${\textstyle \{x_{t}\}}$ beginning at time ${\textstyle t=0}$, and the output of the exponential smoothing algorithm is commonly written as ${\textstyle \{s_{t}\}}$, which may be regarded as a best estimate of what the next value of ${\textstyle x}$ will be. When the sequence of observations begins at time ${\textstyle t=0}$, the simplest form of exponential smoothing is given by the following formulas: ^[1]

${\displaystyle {\begin{aligned}s_{0}&=x_{0}\\s_{t}&=\alpha x_{t}+(1-\alpha )s_{t-1},\quad t>0\end{aligned}}}$

where ${\textstyle \alpha }$ is the smoothing factor, and ${\textstyle 0<\alpha <1}$. If ${\textstyle s_{t-1}}$ is substituted into ${\textstyle s_{t}}$ continuously so that the formula of ${\textstyle s_{t}}$ is fully expressed in terms of ${\textstyle \{x_{t}\}}$, then exponentially decaying weighting factors on each raw data ${\textstyle x_{t}}$ is revealed, showing how exponential smoothing is named.

The simple exponential smoothing is not able to predict what would be observed at ${\textstyle t+m}$ based on the raw data up to ${\textstyle t}$, while the double exponential smoothing and triple exponential smoothing can be used for the prediction due to the presence of ${\displaystyle b_{t}}$ as the sequence of best estimates of the linear trend.

Basic (simple) exponential smoothing

The use of the exponential window function is first attributed to Poisson ^[2] as an extension of a numerical analysis technique from the 17th century, and later adopted by the signal processing community in the 1940s. Here, exponential smoothing is the application of the exponential, or Poisson, window function. Exponential smoothing was first suggested in the statistical literature without citation to previous work by Robert Goodell Brown in 1956, ^[3] and then expanded by Charles C. Holt in 1957. ^[4] The formulation below, which is the one commonly used, is attributed to Brown and is known as "Brown’s simple exponential smoothing". ^[5] All the methods of Holt, Winters and Brown may be seen as a simple application of recursive filtering, first found in the 1940s ^[2] to convert finite impulse response (FIR) filters to infinite impulse response filters.

The simplest form of exponential smoothing is given by the following formula:

${\displaystyle s_{t}=\alpha x_{t}+(1-\alpha )s_{t-1}=s_{t-1}+\alpha (x_{t}-s_{t-1}),}$

where ${\displaystyle \alpha }$ is the smoothing factor, and ${\displaystyle 0\leq \alpha \leq 1}$. In other words, the smoothed statistic ${\displaystyle s_{t}}$ is a simple weighted average of the current observation ${\displaystyle x_{t}}$ and the previous smoothed statistic ${\displaystyle s_{t-1}}$. Simple exponential smoothing is easily applied, and it produces a smoothed statistic as soon as two observations are available. The term smoothing factor applied to ${\displaystyle \alpha }$ here is something of a misnomer, as larger values of ${\displaystyle \alpha }$ actually reduce the level of smoothing, and in the limiting case with ${\displaystyle \alpha }$ = 1 the smoothing output series is just the current observation. Values of ${\displaystyle \alpha }$ close to 1 have less of a smoothing effect and give greater weight to recent changes in the data, while values of ${\displaystyle \alpha }$ closer to 0 have a greater smoothing effect and are less responsive to recent changes. In the limiting case with ${\displaystyle \alpha }$ = 0, the output series is just flat or a constant as the observation ${\textstyle x_{0}}$ at the beginning of the smoothening process ${\textstyle t=0}$.

There is no formally correct procedure for choosing ${\displaystyle \alpha }$. Sometimes the statistician's judgment is used to choose an appropriate factor. Alternatively, a statistical technique may be used to optimize the value of ${\displaystyle \alpha }$. For example, the method of least squares might be used to determine the value of ${\displaystyle \alpha }$ for which the sum of the quantities ${\displaystyle (s_{t}-x_{t+1})^{2}}$ is minimized. ^[6]

Unlike some other smoothing methods, such as the simple moving average, this technique does not require any minimum number of observations to be made before it begins to produce results. In practice, however, a "good average" will not be achieved until several samples have been averaged together; for example, a constant signal will take approximately ${\displaystyle 3/\alpha }$ stages to reach 95% of the actual value. To accurately reconstruct the original signal without information loss, all stages of the exponential moving average must also be available, because older samples decay in weight exponentially. This is in contrast to a simple moving average, in which some samples can be skipped without as much loss of information due to the constant weighting of samples within the average. If a known number of samples will be missed, one can adjust a weighted average for this as well, by giving equal weight to the new sample and all those to be skipped.

This simple form of exponential smoothing is also known as an exponentially weighted moving average (EWMA). Technically it can also be classified as an autoregressive integrated moving average (ARIMA) (0,1,1) model with no constant term. ^[7]

Double exponential smoothing (Holt linear)

Simple exponential smoothing does not do well when there is a trend in the data. ^[1] In such situations, several methods were devised under the name "double exponential smoothing" or "second-order exponential smoothing," which is the recursive application of an exponential filter twice, thus being termed "double exponential smoothing". This nomenclature is similar to quadruple exponential smoothing, which also references its recursion depth. ^[12] The basic idea behind double exponential smoothing is to introduce a term to take into account the possibility of a series exhibiting some form of trend. This slope component is itself updated via exponential smoothing.

One method, works as follows: ^[13]

Again, the raw data sequence of observations is represented by ${\displaystyle x_{t}}$, beginning at time ${\displaystyle t=0}$. We use ${\displaystyle s_{t}}$ to represent the smoothed value for time ${\displaystyle t}$, and ${\displaystyle b_{t}}$ is our best estimate of the trend at time ${\displaystyle t}$. The output of the algorithm is now written as ${\displaystyle F_{t+m}}$, an estimate of the value of ${\displaystyle x_{t+m}}$ at time ${\displaystyle m>0}$ based on the raw data up to time ${\displaystyle t}$. Double exponential smoothing is given by the formulas

${\displaystyle {\begin{aligned}s_{0}&=x_{0}\\b_{0}&=x_{1}-x_{0}\\\end{aligned}}}$,

and for ${\displaystyle t>0}$ by

${\displaystyle {\begin{aligned}s_{t}&=\alpha x_{t}+(1-\alpha )(s_{t-1}+b_{t-1})\\b_{t}&=\beta (s_{t}-s_{t-1})+(1-\beta )b_{t-1}\\\end{aligned}}}$

where ${\displaystyle \alpha }$ (${\displaystyle 0\leq \alpha \leq 1}$) is the data smoothing factor, and ${\displaystyle \beta }$ (${\displaystyle 0\leq \beta \leq 1}$) is the trend smoothing factor.

To forecast beyond ${\displaystyle x_{t}}$ is given by the following approximation:

${\displaystyle F_{t+m}=s_{t}+m\cdot b_{t}}$.

Setting the initial value ${\displaystyle b}$ is a matter of preference. An option other than the one listed above is ${\textstyle {\frac {x_{n}-x_{0}}{n}}}$ for some ${\displaystyle n}$.

Note that F₀ is undefined (there is no estimation for time 0), and according to the definition F₁=s₀+b₀, which is well defined, thus further values can be evaluated.

A second method, referred to as either Brown's linear exponential smoothing (LES) or Brown's double exponential smoothing works as follows. ^[14]

${\displaystyle {\begin{aligned}s'_{0}&=x_{0}\\s''_{0}&=x_{0}\\s'_{t}&=\alpha x_{t}+(1-\alpha )s'_{t-1}\\s''_{t}&=\alpha s'_{t}+(1-\alpha )s''_{t-1}\\F_{t+m}&=a_{t}+mb_{t},\end{aligned}}}$

where a_t, the estimated level at time t, and b_t, the estimated trend at time t, are given by

${\displaystyle {\begin{aligned}a_{t}&=2s'_{t}-s''_{t}\\[5pt]b_{t}&={\frac {\alpha }{1-\alpha }}(s'_{t}-s''_{t}).\end{aligned}}}$

Triple exponential smoothing (Holt Winters)

Triple exponential smoothing applies exponential smoothing three times, which is commonly used when there are three high frequency signals to be removed from a time series under study. There are different types of seasonality: 'multiplicative' and 'additive' in nature, much like addition and multiplication are basic operations in mathematics.

If every month of December we sell 10,000 more apartments than we do in November the seasonality is additive in nature. However, if we sell 10% more apartments in the summer months than we do in the winter months the seasonality is multiplicative in nature. Multiplicative seasonality can be represented as a constant factor, not an absolute amount. ^[15]

Triple exponential smoothing was first suggested by Holt's student, Peter Winters, in 1960 after reading a signal processing book from the 1940s on exponential smoothing. ^[16] Holt's novel idea was to repeat filtering an odd number of times greater than 1 and less than 5, which was popular with scholars of previous eras. ^[16] While recursive filtering had been used previously, it was applied twice and four times to coincide with the Hadamard conjecture, while triple application required more than double the operations of singular convolution. The use of a triple application is considered a rule of thumb technique, rather than one based on theoretical foundations and has often been over-emphasized by practitioners. Suppose we have a sequence of observations ${\displaystyle x_{t},}$ beginning at time ${\displaystyle t=0}$ with a cycle of seasonal change of length ${\displaystyle L}$.

The method calculates a trend line for the data as well as seasonal indices that weight the values in the trend line based on where that time point falls in the cycle of length ${\displaystyle L}$.

Let ${\displaystyle s_{t}}$ represent the smoothed value of the constant part for time ${\displaystyle t}$, ${\displaystyle b_{t}}$ is the sequence of best estimates of the linear trend that are superimposed on the seasonal changes, and ${\displaystyle c_{t}}$ is the sequence of seasonal correction factors. We wish to estimate ${\displaystyle c_{t}}$ at every time ${\displaystyle t}$mod ${\displaystyle L}$ in the cycle that the observations take on. As a rule of thumb, a minimum of two full seasons (or ${\displaystyle 2L}$ periods) of historical data is needed to initialize a set of seasonal factors.

The output of the algorithm is again written as ${\displaystyle F_{t+m}}$, an estimate of the value of ${\displaystyle x_{t+m}}$ at time ${\displaystyle t+m>0}$ based on the raw data up to time ${\displaystyle t}$. Triple exponential smoothing with multiplicative seasonality is given by the formulas ^[1]

${\displaystyle {\begin{aligned}s_{0}&=x_{0}\\[5pt]s_{t}&=\alpha {\frac {x_{t}}{c_{t-L}}}+(1-\alpha )(s_{t-1}+b_{t-1})\\[5pt]b_{t}&=\beta (s_{t}-s_{t-1})+(1-\beta )b_{t-1}\\[5pt]c_{t}&=\gamma {\frac {x_{t}}{s_{t}}}+(1-\gamma )c_{t-L}\\[5pt]F_{t+m}&=(s_{t}+mb_{t})c_{t-L+1+(m-1){\bmod {L}}},\end{aligned}}}$

where ${\displaystyle \alpha }$ (${\displaystyle 0\leq \alpha \leq 1}$) is the data smoothing factor, ${\displaystyle \beta }$ (${\displaystyle 0\leq \beta \leq 1}$) is the trend smoothing factor, and ${\displaystyle \gamma }$ (${\displaystyle 0\leq \gamma \leq 1}$) is the seasonal change smoothing factor.

The general formula for the initial trend estimate ${\displaystyle b}$ is

${\displaystyle {\begin{aligned}b_{0}&={\frac {1}{L}}\left({\frac {x_{L+1}-x_{1}}{L}}+{\frac {x_{L+2}-x_{2}}{L}}+\cdots +{\frac {x_{L+L}-x_{L}}{L}}\right)\end{aligned}}}$.

Setting the initial estimates for the seasonal indices ${\displaystyle c_{i}}$ for ${\displaystyle i=1,2,\ldots ,L}$ is a bit more involved. If ${\displaystyle N}$ is the number of complete cycles present in your data, then

${\displaystyle c_{i}={\frac {1}{N}}\sum _{j=1}^{N}{\frac {x_{L(j-1)+i}}{A_{j}}}\quad {\text{for }}i=1,2,\ldots ,L}$

where

${\displaystyle A_{j}={\frac {\sum _{i=1}^{L}x_{L(j-1)+i}}{L}}\quad {\text{for }}j=1,2,\ldots ,N}$.

Note that ${\displaystyle A_{j}}$ is the average value of ${\displaystyle x}$ in the ${\displaystyle j^{\text{th}}}$ cycle of your data.

Triple exponential smoothing with additive seasonality is given by

${\displaystyle {\begin{aligned}s_{0}&=x_{0}\\s_{t}&=\alpha (x_{t}-c_{t-L})+(1-\alpha )(s_{t-1}+b_{t-1})\\b_{t}&=\beta (s_{t}-s_{t-1})+(1-\beta )b_{t-1}\\c_{t}&=\gamma (x_{t}-s_{t-1}-b_{t-1})+(1-\gamma )c_{t-L}\\F_{t+m}&=s_{t}+mb_{t}+c_{t-L+1+(m-1){\bmod {L}}}.\\\end{aligned}}}$

Implementations in statistics packages

• R: the HoltWinters function in the stats package ^[17] and ets function in the forecast package ^[18] (a more complete implementation, generally resulting in a better performance ^[19] ).
• Python: the holtwinters module of the statsmodels package allow for simple, double and triple exponential smoothing.
• IBM SPSS includes Simple, Simple Seasonal, Holt's Linear Trend, Brown's Linear Trend, Damped Trend, Winters' Additive, and Winters' Multiplicative in the Time-Series modeling procedure within its Statistics and Modeler statistical packages. The default Expert Modeler feature evaluates all seven exponential smoothing models and ARIMA models with a range of nonseasonal and seasonal p, d, and q values, and selects the model with the lowest Bayesian Information Criterion statistic.
• Stata: tssmooth command ^[20]
• LibreOffice 5.2 ^[21]
• Microsoft Excel 2016 ^[22]

See also

• Autoregressive moving average model (ARMA)
• Errors and residuals in statistics
• Moving average
• Continued fraction

Notes

External links

• Lecture notes on exponential smoothing (Robert Nau, Duke University)
• Data Smoothing by Jon McLoone, The Wolfram Demonstrations Project
• The Holt–Winters Approach to Exponential Smoothing: 50 Years Old and Going Strong by Paul Goodwin (2010) Foresight: The International Journal of Applied Forecasting
• Algorithms for Unevenly Spaced Time Series: Moving Averages and Other Rolling Operators by Andreas Eckner