Multivariate GARCH Models: Software Choice and Estimation Issues
Multivariate GARCH Models: Software Choice and Estimation Issues
Multivariate GARCH Models: Software Choice and Estimation Issues
t t
S S
and f
t
= ) / ( 100
1
t t
F F in the usual fashion.
The conditional mean equations for the model that we estimate can be written as
t t
Y + = ,
t
N(0,H
t
) (2)
2
Since Brooks, Henry and Persand (2002) estimated only BEKK models, and this paper uses the diagonal
VECH representation, our results are not directly comparable with theirs.
3
Since these contracts expire 4 times per year - March, June, September and December - to obtain a
continuous time series we use the closest to maturity contract unless the next closest has greater volume, in
which case we switch to this contract. Extensive further details of the data can be found in Brooks, Henry
and Persand.
ISMA Centre Discussion Papers in Finance DP2003-07
where
=
t
t
t
f
s
Y , M is a 2 1 vector of intercepts in the conditional mean (
=
f
s
),
and with the conditional variance-covariance equations being given by (1) using diagonal
forms for A and B. The conditional variance-covariance matrix, H
t
, will comprise the
elements h
s,t
and h
f,t
on the leading diagonal and h
s,f,t
as both of the off-diagonal terms.
For clarity, the conditional mean equations can be written out separately as
t f f t
t s s t
f
s
,
,
+ =
+ =
(3)
with the conditional variance and covariance equations as
1 , , 3 1 , 1 , 3 3 , ,
1 , 2
2
1 , 2 2 ,
1 , 1
2
1 , 1 1 ,
+ + =
+ + =
+ + =
t f s t f t s t f s
t f t f t f
t s t s t s
h b a c h
h b a c h
h b a c h
(4)
The purchase or sale of futures contracts provides a method for hedging exposures to
movements in the price of the underlying asset. In the present context, estimating an
optimal hedge ratio would involve determining the optimal number of futures contracts
that should be sold per holding of the spot asset. Many studies have compared the
performance of time-varying hedge ratios estimated using multivariate GARCH models
with those of nave or time-invariant hedge ratios estimated using OLS regressions. The
majority of these studies have preferred the time-varying approach (see, for example,
Baillie and Myers, 1991) on the grounds that they provide slightly more accurate hedge
ratio estimation leading to portfolio returns with lower variances. Given the coefficients
and fitted values from the estimated model, it is possible to show that the optimal hedge
ratio will be given by the negative of the ratio of the one-step ahead forecast of the
covariance between the spot and futures returns to the one-step ahead forecast of the
futures return variance:
ISMA Centre Discussion Papers in Finance DP2003-07
t f
t f s
t
h
h
,
, , *
1
=
(5)
When the hedge ratio is expressed in this way, the returns to the hedged portfolio can be
written as
t t t t p
f s r
*
1 ,
+ =
(6)
It is also possible to express the variance of the returns to the hedged portfolio as
t f s t t f t t s t p
h h h r
, ,
*
1 ,
2
*
1 , ,
2 ) var(
+ = (7)
3. The packages
3.1 Background
Brooks, Burke and Persand (2001) evaluated 9 packages for the estimation of univariate
GARCH models. Of these 9, only 4 contain pre-programmed routines for the estimation
of multivariate GARCH models: EViews, GAUSS, RATS and SAS. Thus, multivariate
GARCH models cannot be estimated using the currently available versions of LIMDEP,
MATLAB, MICROFIT, SHAZAM, or TSP
4
. In addition, whilst the current version of
EViews (4.0) incorporates sample routines for estimating the BEKK formulation, it does
not include similar instructions for estimating a diagonal VECH model. Even though
code for estimating the latter model could be obtained by making relatively trivial
modifications to the former, we chose not to include EViews in the review, since the
resulting assessment would be a joint one of EViews estimation of the VECH model and
our programming skills in that package.
4
Of course, provided that the package incorporates some sort of programming capability for users, and that
it is possible to manipulate the maximum-likelihood estimation routines, a skilled programmer may be able
to set up the model and estimate it herself. This may be possible with, for example, MATLAB (although
multivariate GARCH models have not been already coded into the MATLAB GARCH toolbox), although
it would prove impossible for a pure click-and-point package such as MICROFIT.
ISMA Centre Discussion Papers in Finance DP2003-07
Given the widespread use of this class of models, and that they are now more than a
decade and a half old, it is rather surprising that more developers have not included
routines to estimate such models in their packages. For any package that contains a
maximum likelihood optimiser, an extension to allow for MGARCH models would not
be a difficult exercise. In addition to the packages employed by Brooks et al. (2001) that
allow for MGARCH model estimation, this review also considers the FINMETRICS
add-in module for S-PLUS
5
. Other packages, including PC-GIVE and STATA were
investigated, but these too only included the provision for estimating univariate GARCH
models.
Table 1 presents contact and version details for the four packages. Clearly, a first concern
is whether the package in question is able to estimate the model of interest for a
particular researcher, and therefore the last 4 columns of Table 1 indicate which models
from the list of full VECH, constant correlation, diagonal VECH and BEKK the
packages are able to estimate. It turns out that most of the packages are fairly flexible,
and allow the estimation of at least three of the four types of multivariate GARCH
model. The only exceptions are that the full unrestricted VECH is not available with
FANPAC or FinMetrics and the constant correlation model is not available with SAS
although neither of these probably represent an important loss of functionality in practice.
3.2 Flexibility versus Functionality
Clearly there is an important trade off in practice between flexibility and ease of use. We
would argue that multivariate GARCH formulations are sufficiently complex that those
researchers with no programming ability at all are unlikely to be consumers of such
5
Jean-Philippe Peters and Sebastien Laurent are currently in the process of producing a new version of
their G@RCH add-in for OX, and it is understood that their new version will include the capability to
estimate multivariate GARCH models see www.egss.ulg.ac.be/garch .
ISMA Centre Discussion Papers in Finance DP2003-07
models, and therefore that the range of estimable models and the range of estimation
options available are likely to be more important criteria for determining the usefulness
of the software than how many buttons must be pressed before some results are obtained.
An important question in practice, therefore, is whether the researcher can get at the
likelihood object in other words, can the user add exogenous variables into the
conditional variance or covariance equations or can the user employ an alternative (e.g.,
logarithmic) specification for the equations or employ an alternative distribution for the
underlying disturbances? The answer, subject to the researcher being a sufficiently adept
programmer in the package concerned, is yes for any package where the user specifies
how the equations to be estimated and the log-likelihood function are set up. This would
be the case for RATS, where an exogenous variable could simply be added to the desired
equation. But the range of estimable models is much more limited for GAUSS-FANPAC,
SAS, or for S-Plus FinMetrics
6
, where the researcher simply calls a sub-routine that is
hard-coded and into which no access is granted. The latter packages of course therefore
entail a much more compact set of instructions to estimate the model approximately 13
and 15 lines respectively for GAUSS and SAS compared to perhaps double that for
RATS. Once the data are loaded into memory, the estimation in S-PLUS FINMETRICS
can be performed in one line, making it by far the most compact set of code.
3.3 Speed and Documentation
Given the computer power that is now widely available, the speed at which models are
estimated is scarcely an issue worth mentioning in a software review unless one is
6
FINMETRICS does permit the user to select t-distributed disturbances instead of Gaussian, and to add
additional variables into the conditional mean or variance equations, and to employ higher-order terms in
the conditional variance or covariance equations. Therefore, it does offer a considerable degree of
flexibility, but less than the complete control users can obtain from RATS.
ISMA Centre Discussion Papers in Finance DP2003-07
conducting a Monte Carlo study where such models must be estimated tens of thousands
of times. For the 4 packages considered here, there was little to choose between them in
terms of the time taken to estimate the models - typically 1 or 2 minutes were required on
a Pentium II 333 MHz P.C. with 196 Mb RAM and running Windows 98
7
.
The documentation related to the estimation of multivariate GARCH models for each of
the packages is adequate; ideally help should be available on-line as well as in hard copy
form. Arguably, GAUSS FANPAC and S-PLUS FINMETRICS provide the most
extensive written documentation on this particular class of models, and the maximum
likelihood routine is also well described in the RATS manual. SAS provides less written
documentation on the operation of that particular part of the software, which is somewhat
disappointing given that the combined SAS manuals run to several thousand pages.
However, substantially more detail on PROC VARMAX is available on-line
8
.
4. Model Estimation and Results
We estimate the model parameters using as close to the default settings as possible with
each package. There are two reasons for doing this. First, anecdotal evidence suggests
that many researchers simply employ the default settings on the grounds of simplicity
without examining whether they are optimal. Exclusive use of default settings can also
occur as a result of the researchers lack of knowledge of the details of the package or of
the technical details of how the estimation actually operates. Second, to the extent that
any one approach to estimation can be considered generally superior to others, it is
reasonable to assume that the developers would make the default model estimation
routines the ones that are likely to be the most reliable or robust, rather than hiding them
7
All packages were run on the same computer and Windows platform to ensure consistency.
8
http://v9.doc.sas.com/
ISMA Centre Discussion Papers in Finance DP2003-07
in a footnote in the manual. Additionally, Fiorentini et al. (1996) demonstrated via a
Monte Carlo study that in the context of univariate GARCH model estimation, increased
accuracy results from the use of analytic gradients and Hessian than from approaches
based only on numerical approximations. Analytic information is used in computing the
derivates when estimating univariate GARCH models by GAUSS and SAS but not by
RATS or S-PLUS, whilst analytic information is not used in construction of the Hessian
under any of the packages; only numerical procedures are used for computing the
derivatives when estimating multivariate GARCH models under all packages.
Table 2 shows the results from estimating the bivariate GARCH model using the spot
and futures returns described above. The parameter estimates are shown to 3 decimal
places and the asymptotic t-ratios to 2. An interesting side-issue is the considerable
variation in the apparent precision with which these numbers are reported: GAUSS-
FANPAC only reports to 3 decimal places, SAS and S-PLUS FINMETRICS to 5, and
RATS to 9.
The default estimations under SAS failed, and once this happens, there is no unique way
to proceed. The SAS developers have stated that the PROC VARMAX procedure is
experimental rather than production in version 8.2, as well in versions 9 and the
forthcoming 9.1. SAS estimation resulted in a non-positive definite variance-covariance
matrix, but a switch from the default optimisation to the Quasi-Newton approach using
the nloptions tech=quanew; instruction for SAS results in plausible parameter and
standard error estimates
9
. Default estimation using GAUSS-FANPAC, RATS and S-
9
The SAS developers have recommended the use of the UDP=DDFP and MAXFUNC=6000
specifications for this data and model. This will estimate the model using quasi-Newton optimisation with
the dual Davidson Fletcher Powell (DFP) update of the Cholesky factor of the Hessian matrix with the
maximum possible number of function calls raised to 6000.
ISMA Centre Discussion Papers in Finance DP2003-07
PLUS FINMETRICS results in plausible parameter and standard error estimates without
user intervention
10
. Note that, in the absence of a benchmark with which to compare the
estimated parameters and their standard errors, it is really impossible to say any more
about them other than to assess in a qualitative sense whether they seem sensible given
the results of existing studies using similar models.
Examining first the parameter estimation, the degree of variation between the packages is
both surprising and potentially worrying. The intercepts in the conditional mean
equations are similar for GAUSS, RATS and SAS, but are almost a third higher for S-
PLUS. However, it is the conditional variance and covariance equations where the
differences across packages become marked. The intercept in the spot (cash) conditional
variance equation (c
1
) is around 0.01 for RATS and SAS, but 0.08 for S-PLUS and 0.4
for GAUSS a 40-fold gap between the highest and lowest estimate. An even bigger
divergence occurs with the estimates for the same parameter in the futures conditional
variance equation and in the covariance equation (c
2
and c
3
respectively). The parameters
on the lagged squared errors (a
1
and a
2
) are also higher for GAUSS and S-PLUS than for
RATS or SAS, but this time only by a factor of around 4. Finally, the parameter
estimates for the lagged conditional variances and covariance are again close for RATS
and SAS at around 0.95, whereas they are around 0.4 for GAUSS and 0.8 for S-PLUS.
In some senses, GAUSS is the odd one out, spreading the weight in the measure of
persistence equally on h
t
and
t-1
2
, whereas the other three packages give much bigger
estimates on h
t
than on
t-1
2
. Interestingly, the variation in estimation of the same
parameter across packages is far greater than the variation in estimation for the same
parameter across equations for a given package. This may arise from the tendency for a
10
Note that by plausible, all we mean is that the parameter estimates in the conditional variance
equations are positive and non-explosive, and that the standard errors are also positive.
ISMA Centre Discussion Papers in Finance DP2003-07
given package to use the same set of initial estimates for the parameters on the lagged
squared error and lagged conditional variance/covariance for all equations.
Turning now to the standard error estimation, the results of which are given in the second
panel of Table 2, and the t-ratios given in the third panel, it is evident that the differences
across packages are even more marked than they were for the parameter estimates. The t-
ratios for SAS are considerably larger than those of the other packages for all of the
parameters, resulting from SASs orders of magnitude smaller estimates of the standard
errors. Most notably, the SAS t-ratios are around 100 times higher than the next highest
set for the intercept in the conditional mean spot equation and for the parameter on the
lagged futures conditional variance. However, none of these differences are important for
tests of significance: given the large sample size, all of the parameters are statistically
significant at the 0.1% level under all packages.
The differences in standard error estimation are arguably unsurprising since a similar
result was found by Brooks et al. (2001) in the context of the estimation of simpler
univariate GARCH models. But the differences in parameter estimation are substantial,
and this result is quite in contrast with Brooks et al., who found only modest differences
across software. Multivariate models, by their very nature, are inherently more complex
to estimate than their univariate counterparts, and this considerably increases the scope
for the optimisation routine to run into problems: for example, to find only a local
optimum or not to converge at all. Two obvious questions arise from these results: First,
why are the parameter estimates so very different, and second, does it matter? The first of
these questions could probably be answered by examining the differences in optimisation
technique across packages. Differences could arise in the default settings according to the
optimisation routine used (e.g. BHHH versus Newton), the use of analytic or numerical
ISMA Centre Discussion Papers in Finance DP2003-07
derivatives, differences in initialisations for the error and conditional variance /
covariance series, differences in parameter initial estimates, or differences in
convergence criteria. A thorough examination of all of these issues is virtually impossible
since the packages on the whole simply do not give sufficient detail on these points.
Ideally, a package would give as much flexibility as possible for users to specify the
optimisation controls, and arguably the best package in this regard is RATS. Only RATS
gives the opportunity for the user to modify all of the controls in the list above. In terms
of optimisation routine, GAUSS and SAS use a version of BFGS whereas S-PLUS uses
BHHH with no opportunity to use an alternative approach. GAUSS does not allow
modification of the convergence criteria, the initialisations of the error and
variance/covariance series or the starting values for the parameter estimates. In terms of
the methods that can be used to calculate standard errors, a method based on the Hessian
(default) or QML are available with GAUSS, the Hessian (default), OPG or QMLE are
available with RATS, the Hessian only is available with SAS, while Hessian, OPG
(default) and QMLE are available with S-PLUS.
Now addressing the issue of whether the differences in parameter estimation between
packages makes a difference from a practical perspective, we calculate the (in-sample)
time-varying hedge ratios using equation (5) above together with the series of fitted
conditional variances and covariances for each package. Unfortunately, it is not possible
to use SAS to perform this calculation since the current version of the PROC
VARMAX procedure does not permit the user to output the fitted conditional variances
or covariances. The optimal hedge ratios (OHRs) calculated in this fashion for the
remaining three packages are plotted in Figure 1. Given the in some cases enormous
differences in parameter estimation, the profiles of the OHRs are quite similar, although
ISMA Centre Discussion Papers in Finance DP2003-07
there is clear evidence of it being considerably more variable for GAUSS and S-PLUS
than for RATS. On the whole, however, the OHRs are rather unstable, ranging from
below 0.4 to above 1, and thus any firm attempting to use an MGARCH model for this
purpose would face substantial rebalancing costs. This range compares with a time-
invariant OHR calculated using OLS (in Microsoft Excel) of 0.80.
Finally, given that OHRs have been constructed using each of the packages, it is
possible to examine how much protection these would have offered a firm in terms of
reduced portfolio volatility, measured by the standard deviation of portfolio returns.
These results are presented in Table 3, together with those arising from the use of the
time-invariant OLS hedge and from using no hedge at all. The mean of the portfolio
returns, calculated using equation (6) is not of direct interest since the objective of
hedging is to reduce volatility and not to increase returns. Remarkably, in spite of the
enormous differences in parameter estimates, the standard deviations of portfolio returns
(calculated by taking the square root of equation (7)) are almost identical across the 3
packages (and the OLS hedge). Thus, whilst the benefit from engaging in hedging is
clear, it does not matter which package you use to calculate the OHRs and you are just as
well not to bother with MGARCH models at all but to stick to OLS!
Conclusions
This review has sought to compare and contrast the four packages available for
estimating multivariate GARCH models: GAUSS-FANPAC, RATS, SAS and S-PLUS.
Considerable differences in the resulting parameter estimates were observed, but these
turned out to be relatively unimportant from a practical point of view. But how can this
be the case? The answer appears to lie in the differences between the packages cancelling
out to a large extent, and this cancelling out occurs on two levels. First, estimates of the
ISMA Centre Discussion Papers in Finance DP2003-07
unconditional variances and covariances are much closer across the packages than the
parameter estimates would suggest. For example, the unconditional variances of the spot
returns implied by the model estimates are 0.82 (GAUSS), 0.92 (RATS), 0.77 (SAS), and
0.84 (S-PLUS). Further cancelling out appears to arise when both the conditional
variances and covariances are over-calculated and then the latter is divided by the former
in the construction of the hedge ratio.
To summarise, it is worth reiterating that in the absence of a benchmark dataset and
results, it is not possible to say which set of parameter estimates arising from the various
software packages is best, but clearly prima facie they represent very different
characterisations of the data. There is much work to be done if this class of models is to
be reliably used in practice and we argue that the development of such a benchmark
would be a worthwhile activity. A further implication of our results is the indication that
researchers should focus upon the end use of their model when attempting to evaluate it
and not necessarily on the parameter estimates.
Acknowledgements
The authors are grateful to James MacKinnon and to Kevin Meyer for useful comments
on a previous version of this paper. The authors alone bear responsibility for any
remaining errors.
ISMA Centre Discussion Papers in Finance DP2003-07
References
Baillie, R.T. and Myers, R.J. (1991) Bivariate GARCH Estimation of the Optimal Commodity
Futures Hedge Journal of Applied Econometrics 6, 109-124.
Bollerslev, T. (1990) Modelling the Coherence in Short-Run Nominal Exchange Rates: A
Multivariate Generalised ARCH Model Review of Economics and Statistics 72, 498-505.
Bollerslev, T., Engle, R.F. and Wooldridge, J.M. (1988) A Capital Asset Pricing Model with
Time-varying Covariances Journal of Political Economy 96, 116-31.
Brooks, C. (1997) GARCH Modelling in Finance: A review of the Software Options Economic
Journal 107(443) 1271-1276.
Brooks, C., Burke, S.P. and Persand, G. (2001) Benchmarks and the Accuracy of GARCH Model
Estimation International Journal of Forecasting 17, 45-56.
Brooks, C., Henry, O.T. and Persand, G. (2002) The Effect of Asymmetries on Optimal Hedge
Ratios Journal of Business, 75(2), 333-352.
Engle, R.F. and Kroner, K. (1995) Multivariate Simultaneous Generalised ARCH Econometric
Theory 11, 122-150.
Fiorentini, G., Calzolari, G., and Panattoni, L. (1996) Analytic Derivatives and the Computation
of GARCH Estimates Journal of Applied Econometrics 11, 399-417
Kroner, K.F., and Ng, V.K. (1998) Modelling Asymmetric Co-movements of Asset Returns
Review of Financial Studies 11, 817-844.
McCullough, B.D. and Renfro, C.G. (1999) Benchmarks and Software Standards: A Case Study
of GARCH Procedures Journal of Economic and Social Measurement 25, 59-71.
ISMA Centre Discussion Papers in Finance DP2003-07
Table 1: Details of Packages Employed and the Range of Estimable Multivariate
GARCH Models
Package and
Version used
Contact information
Can the following models be estimated?
Full Constant Diagonal BEKK
VECH Correlation VECH
GAUSS 3.2.39-
FANPAC
1.1.11/2
Aptech Systems Inc, 23804
S.E. Kent, Langley Road, Maple
Vallet, WA 98038, USA
http://www.aptech.com
No
Yes
Yes
Yes
RATS 4.3
Estima, PO Box 1818, Evanston, IL
60204-1818, USA
http://www.estima.com
Yes
Yes
Yes
Yes
SAS 8.2
1
SAS Institute, Campus Drive, Cary
NC 27513 USA.
http://www.sas.com
Yes
No
Yes
Yes
S-PLUS 6.1
FINMETRICS
1.0
Insightful Corporation
1700 Westlake Avenue N, Suite 500
Seattle WA 98109-3044 USA
www.insightful.com
No
Yes
Yes
Yes
1
At the time of writing this review, SAS 8.2 was the most up-to-date version of the software available,
although version 9 is now available. It is possible that the results for the latter version may be quite
different from the former.
ISMA Centre Discussion Papers in Finance DP2003-07
Table 2: Parameter Estimates for Multivairate GARCH Model using FTSE Spot and
Futures Returns: 1 January 1985 - 9 April 1999
Model:
t f f t
t s s t
f
s
,
,
+ =
+ =
,
1 , , 3 1 , 1 , 3 3 , ,
1 , 2
2
1 , 2 2 ,
1 , 1
2
1 , 1 1 ,
+ + =
+ + =
+ + =
t f s t f t s t f s
t f t f t f
t s t s t s
h b a c h
h b a c h
h b a c h
Panel A: Parameter Estimates
Package
c
f
c
1
a
1
b
1
c
2
a
2
b
2
c
3
a
3
b
3
GAUSS 0.064 0.064 0.377 0.128 0.411 0.566 0.145 0.365 0.474 0.128 0.348
RATS 0.062 0.069 0.012 0.041 0.946 0.012 0.034 0.956 0.011 0.035 0.953
SAS 0.061 0.067 0.010 0.037 0.952 0.010 0.031 0.961 0.009 0.032 0.959
S-PLUS 0.073 0.082 0.076 0.112 0.798 0.125 0.134 0.762 0.099 0.120 0.773
Panel B: Standard Error Estimates
Package SE(
c
) SE(
f
) SE(c
1
) SE(a
1
) SE(b
1
) SE(c
2
) SE(a
2
) SE(b
2
) SE(c
3
) SE(a
3
) SE(b
3
)
GAUSS 0.014 0.016 0.030 0.013 0.041 0.044 0.013 0.039 0.014 0.012 0.011
RATS 0.014 0.016 0.001 0.002 0.003 0.001 0.002 0.002 0.001 0.002 0.003
SAS 0.019 0.019 0.073 0.098 0.192 0.008 0.023 0.047 0.018 0.038 0.060
S-PLUS 0.013 0.015 0.005 0.007 0.010 0.009 0.007 0.011 0.007 0.007 0.010
Panel C: Estimated t-ratios
Package t(
c
) t(
f
) t(c
1
) t(a
1
) t(b
1
) t(c
2
) t(a
2
) t(b
2
) t(c
3
) t(a
3
) t(b
3
)
GAUSS 4.57 4.00 12.57 9.85 10.02 12.86 11.15 9.36 33.86 10.67 31.64
RATS 4.51 4.23 9.24 17.00 344.79 9.52 16.25 407.69 9.51 15.96 375.43
SAS 313.91 350.61 13.10 37.65 496.86 126.45 134.55 999.00 52.88 82.96 999.00
S-PLUS 5.68 5.56 14.13 16.52 77.02 14.54 18.00 68.08 15.02 17.84 74.16
Note: The standard errors for SAS have been multiplied by 100 for display in the table.
Table 3: In-Sample Performance of Optimal Portfolios
Package Mean of Portfolio Returns Standard Deviation of
Portfolio Returns
GAUSS MGARCH 0.010 0.357
RATS MGARCH 0.065 0.350
S-PLUS - MGARCH 0.009 0.355
OLS Hedge 0.009 0.348
No hedge 0.046 0.962
ISMA Centre Discussion Papers in Finance DP2003-07
Figure 1: Fitted Optimal Hedge Ratios
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
1
1
1
7
2
3
3
3
4
9
4
6
5
5
8
1
6
9
7
8
1
3
9
2
9
1
0
4
5
1
1
6
1
1
2
7
7
1
3
9
3
1
5
0
9
1
6
2
5
1
7
4
1
1
8
5
7
1
9
7
3
2
0
8
9
2
2
0
5
2
3
2
1
2
4
3
7
2
5
5
3
2
6
6
9
2
7
8
5
2
9
0
1
3
0
1
7
3
1
3
3
3
2
4
9
3
3
6
5
3
4
8
1
Observation
H
e
d
g
e
R
a
t
i
o
RATS
GAUSS
S-PLUS