Quantitative Finance

ARMA processes

Jesús Ramirez

Universidad del Pacífico

Table of contents

1. ARMA processes

2. Testing Sample Autocorrelation

3. R2 and ARMA processes

4. Estimation

1
ARMA processes
Autoregressive Processes

AR processes

• A pth -order AR process, or AR(p) process:

Yt = ϕ0 + ϕ1 Yt−1 + ϕ2 Yt−2 + ... + ϕp Yt−p + εt

where ε ∼ WN(0, σ 2 )

2
Moving Average Processes

MA processes

• A qth -order MA process, or MA(q) process:

Yt = ϕ0 + εt + θ1 εt−1 + ... + θq εt−q

where ε ∼ WN(0, σ 2 )

• MA(1) can be written as a weighted average of all lags of Yt plus

an innovation term.

3
Autoregressive Moving Average Processes

ARMA processes

• An ARMA(p,q) process process:

Yt = ϕ0 + ϕ1 Yt−1 + ϕ2 Yt−2 + ... + ϕp Yt−p

+εt + θ1 εt−1 + θ2 εt−2 + ... + θq εt−q

where ε ∼ WN(0, σ 2 )

• Usually, we set p and q between 0 and 2, to have a small

number of parameter to estimate.

4
Partial Autocorrelation Function

Definition
Let Yt be a AR process, the partial autocorrelation between Yt and
Yt−s is the autocorrelation that eliminates the effects of the
intermediate values Yt−1 through Yt−s+1 .

5
How to compute the PACF

Procedure

1. Define a new variable Y∗t = Yt − µ.

2. Form the first-order autoregression:

Y∗t = ϕ11 Y∗t−1 + et .

the ϕ11 is the PACF between Yt and Yt−1

3. Then, form the second-order autoregression:

Y∗t = ϕ21 Y∗t−1 + ϕ22 Y∗t−2 + et

the ϕ22 is the PACF between Yt and Yt−2

4. What is next? I think, you will figure it out by yourself.

6
Plotting the ACF and PACF

Fuente: Enders, W., Applied Econometric Time Series, 4th Edition

Figure 1: ACF and PACF

7
Properties of the ACF and PACF

Fuente: Enders, W., Applied Econometric Time Series, 4th Edition

Figure 2: ACF and PACF properties

8
Testing Sample Autocorrelation
Sample Autocorrelation

• In practice, mean, variance and autocorrelation are unknown to

the researcher.

Sample mean, variance and autocorrelation

Let there be T observations labeled y1 through yT and let ȳ, σ̂ 2 and
ρ̂j be estimates of µ, σ 2 and ρj respectively, where

1∑ 1∑
T T
ȳ = yt , σ̂ 2 = (yt − ȳ)2
T t=1 T t=1

∑T
t=j+1 (yt − ȳ)(yt−j − ȳ)
1
T−j
ρ̂j = ∑T
T
1
t=1 (yt − ȳ)2

9
Test on Individual Autocorrelation Coefficients

Bartlett t-statistic
iid
If data ∼ N (µ, σ 2 ), then the asymptotic distribution of ρ̂ is
( )
1
ρ̂j ∼ N 0,
T

Equivalently, we can construct a t-statistic for ρ̂j

ρ̂j √
tρ̂j = = Tρ̂j ∼ N (0, 1)
s.e.(ρ̂j )

10
Test on Individual Autocorrelation Coefficients

Robust t-statistic

• If data exhibit heteroskedasticity, then it is preferable to use a

”robust” estimator for the standard errors.
• You can compute the robust t-statistic by running the
regression:

Yt = β0 + β1 Yt−j + et

and test H0 : βj = 0 vs H1 : βj ̸= 0, using robust standard errors

for the parameter estimated.

• Confidence intervals for each autocorrelation need not to be the

same.

11
Joint Test on many Autocorrelation

Ljung-Box Q-statistic
iid
If data ∼ N (µ, σ 2 ), we can test:

H0 : ρ1 = ρ2 = ... = ρL = 0 vs H1 : ρj ̸= 0, j = 1, 2, ..., L

using the Ljung-Box Q-statistic, QLB (L), given by:

∑
L
1 2
QLB (L) = T(T + 2) ρ̂ ∼ χ2L
T−j j
j=1

• On daily data, it is recommended to use 10 lags (maybe 20). For

weekly, 4 to 8 lags, and for monthly, 6 to 12 lags.

12
Joint Test on many Autocorrelation

Robust approach
You can test for serial correlation up to given lag L by estimating
the following regression:

Yt = β0 + β1 Yt−1 + β2 Yt−2 + ... + βL Yt−L

and obtaining the robust standard errors for the parameter

estimated. Then test the joint hypothesis:

H0 : β1 = β2 = ... = βL = 0 vs H1 : βj ̸= 0, j = 1, 2, ..., L

using a χ2L test.

13
R2 and ARMA processes
Predictability

From Econometric 1, we know that R2 measures the proportion of the

variation in the dependent variable explained by the model

Var(εt+1 )
R2 = 1 −
Var(Yt+1 )

Challenge: Example 1
Let Y1 be an AR(1) process defined as Yt+1 = ϕ0 + ϕ1 Yt + εt+1 , where
εt+1 ∼ WN(0, σ 2 ). Compute the R2 from this model.

14
Challenge

Example 2
Let Yt and Xt be two AR(1) and MA(1) processes where

Yt = ϕYt−1 + εt , |ϕ| < 1

Xt = θut−1 + ut

where εt ∼ N (0, 1) and ut ∼ N (0, σ 2 )

1. Find the variance of the forecast error for the optimal one-step
and two-step forescast of each Yt and Xt
2. Find the values of θ and σ 2 that makes Yt and Xt equally
predictable (according to the variance of the forecast errors) for
the one-step and two-step ahead forecast.
3. Given these values, which variable is easier to predict
three-steps ahead?

15
Choosing the best ARMA model

Mean Squared Error (MSE)

A very common measure of goodness of fit is the mean squared
error which is defined as:

1∑
T
MSE = et
T t=1

where et is the residual of the ARMA model.

• Lower MSE means model is providing a better fit.

• MSE never increases with the order of the ARMA model.
• However, adding more variables can generate overfitting1 .

1 Overfitting lowers MSE of a model, but ruins its ability for forecasting

16
Goodness of Fit Measures

• MSE ignores the impact of estimation error on forecast accuracy.

• A better measure would be and adjusted-MSE defined as:

1 ∑
T
s= et
T − k t=1

where k is the number of parameters in the regression.

• Other goodness-of-fit measure can be constructed based on the
MSE.

17
Goodness of Fit Measures

Information criteria
The AIC, HQIC and BIC2 , are defined as:

AIC = exp(2k/T) · MSE

HQIC = log(T)2k/T · MSE
√ 2k/T
BIC = T · MSE

• The AIC and BIC are the two most widely-used model selection
criteria.
• The BIC choose smaller models, which is generally a good thing
for forecasting.
• The AIC tends to choose larger models.
2 AIC, HQIC and BIC stand for Akaike Information Criterion, Hannan-Quinn Information
Criterion and Schwartz’s Bayesian Information Criterion, respectively

18
Estimation
ARMA estimation

• AR(p) can be easily estimated using OLS regression on

observation p + 1 through T.
• A useful diagnosis test of the model is to plot the ACF of the
residuals and perform an autocorrelation test on the residuals3 .
• Unlike AR models, MA models must be estimated numerically by
Maximum Likelihood Estimation (MLE).

3 For example, you can perform the Ljung-Box with m − p degrees of freedom

19
MA(1) estimation

Let Yt as MA(1) process:

iid
Yt = θ0 + θ1 εt−1 + εt , where εt ∼ N (0, σε2 )

Procedure

1. Set the unobserved ε0 = 0.

2. Set the parameters starting values4 (initial guess) for θ0 , θ1 and
σε2 .
3. Compute the time series residual via

εt = Yt − θ0 − θ1 εt−1

4 One initial set of parameters could θ0 = Ȳt , θ1 = 0 and σε2 = Var(Yt )

20
MA(1) estimation

iid
• If we assumed that εt ∼ N (0, σε2 ), then its density function is:
( )
1 ε2
f(εt ) = √ exp − t 2
2πσε2 2σε

then, their joint density is:

f(ε1 , ε2 , ..., εT ) = f(ε1 )f(ε2 )...f(εT )

21
MA(1) estimation

Procedure

4. Compute the maximum likelihood function.

∏
T ( )
1 ε2
L(Y1 , Y2 , ..., YT |θ0 , θ1 , σε2 ) = √ exp − t 2
2πσε2 2σε
i=1

where εt = Yt − θ0 − θ1 εt−1 with ε0 = 0

5. Optimize numerically the log-likelihood function, l = ln(L).

T T ∑ ε2 T
t
l(Y1 , Y2 , ..., YT |θ0 , θ1 , σε2 ) = − ln(2π) − ln(σε2 ) −
2 2 2σε2
i=1

22
Estimation Procedure

Simple recipe

1. Look at the sample ACF and sample PACF to identify potential

models that can generate our data.
2. Estimate the potential models and compute the standard errors
of the estimated parameters.
3. Compute the Q-stats and the information criteria (AIC/BIC) for
each model.

23
Programming Challenge

Example 1

1. Simulate 500 data points from the following ARMA(1, 1):

Yt = −0.7Yt−1 + εt − 0.7εt−1

where Y0 and ε0 will be set to zero (In R set the random seed to
my favorite prime number, 2)
2. Now, imagine that you do not know the true data-generating
process and identify 3 potential models for your data.
3. Estimate those models and compute the statistics that are
mentioned in Simple recipe.

24