Torój - OLS Revisited
Torój - OLS Revisited
Torój - OLS Revisited
Andrzej Torój
Outline
1 Introduction
Course information
Econometrics: a reminder
Outline
1 Introduction
Course information
Course information
Course information
Course information
Course information
Course information
Econometrics: a reminder
Why econometrics?
investigation of relationships
finding parameter values in economic models (e.g. elasticities)
confronting economic theories with data
forecasting
simulating policy scenarios
Econometrics: a reminder
Why econometrics?
investigation of relationships
finding parameter values in economic models (e.g. elasticities)
confronting economic theories with data
forecasting
simulating policy scenarios
Econometrics: a reminder
Why econometrics?
investigation of relationships
finding parameter values in economic models (e.g. elasticities)
confronting economic theories with data
forecasting
simulating policy scenarios
Econometrics: a reminder
Why econometrics?
investigation of relationships
finding parameter values in economic models (e.g. elasticities)
confronting economic theories with data
forecasting
simulating policy scenarios
Econometrics: a reminder
Why econometrics?
investigation of relationships
finding parameter values in economic models (e.g. elasticities)
confronting economic theories with data
forecasting
simulating policy scenarios
Econometrics: a reminder
Example (1/4)
Student satisfaction survey
Master students of Applied Econometrics at Warsaw School of Economics in
Winter semester 2016/2017 were asked about their satisfaction from studying
to be evaluated from 0 to 100. In addition, their average note from previous
studies and their sex were registered.
1 What kind of data is this? Cross-section, time series, panel?
Frequency? Micro- or macroeconomic?
2 How can we quickly visualise a hypothesised causality from average
note to satisfaction from studying?
3 Does such a relationship seem to be there?
4 How can sex of the respondent potentially affect the satisfaction from
studies or the relationship in question? How can we visualise this?
5 Bottom line, what is the right specification of the linear regression
model?
Outline
1 Introduction
Point estimation
yi = β0 + β1 x1,i + β2 x2,i
+ ... +
βk xk,i + εi =
β0
β1
1 x1,i x2,i . . . xk,i β2 + εi = xi β + εi
..
.
βk
T
Vector of parameters β0 β1 β2 . . . βk is unknown.
Minimize the dispersion of εi around zero, as measured e.g. by
n
ε2i .
P
t=1
Point estimation
n n
2
ε2i =
P P
S= (yi − β0 − β1 x1,i − β2 x2,i − . . . − βk xk,i ) → min
i=1 i=1 β0 ,β1,...
∂S
FOC: ∂β =0
β0
1
y1 x1,1 x2,1 ... xk,1
β1
y2 1 x1,2 x2,2 ... xk,2
Denote: y=
, X =
, β =
β2
.
. .
. .
. .
. .
.
. . . . ... . .
.
.
yn 1 x1,n x2,n ... xk,n
βk
and obtain:
−1 T
β = XT X X y
Point estimation
Proof
n
ε2i = εT ε = (y − Xβ)T (y − Xβ) =
P
S =
i=1
= − β T XT y − yT Xβ + β T XT Xβ =
yT y
= yT y − 2yT Xβ + β T XT Xβ
(2. and 3. component were transposed scalars, so they were equal)
∂S ∂yT y 2yT Xβ β T XT Xβ
∂β = 0 ⇐⇒ ∂β − ∂β + ∂β =0
According to the rulesof matrix calculus:
T T
−2yT X T T
+ β T2X X = 0 99K β X X = y X 99K
T
T
X X β=X y
−1 T
β = XT X X y
Point estimation
Example (2/4)
Measuring precision
Measuring precision
Measuring precision
Measuring precision
Definition: n o
Var (β) = E [β − E (β)] [β − E (β)]T
For a centered variable, i.e. E (ε) = 0, this definition simplifies:
T
Var (ε) = E εε
Measuring precision
−1 T
β̂ = XT X X y is an estimator (function of the sample) of
the “true”, unknown values β (population / data generating
process). Under certain conditions (i.a. E (X T ε) = 0
E (εεT ) = σ 2 I ), the OLS estimator is:
unbiased: E β̂ = β
consistent: β̂ converges to β with growing n
efficient: least possible estimator variance (i.e. highest
precision)
Measuring precision
−1 T
β̂ = XT X X y is an estimator (function of the sample) of
the “true”, unknown values β (population / data generating
process). Under certain conditions (i.a. E (X T ε) = 0
E (εεT ) = σ 2 I ), the OLS estimator is:
unbiased: E β̂ = β
consistent: β̂ converges to β with growing n
efficient: least possible estimator variance (i.e. highest
precision)
Measuring precision
−1 T
β̂ = XT X X y is an estimator (function of the sample) of
the “true”, unknown values β (population / data generating
process). Under certain conditions (i.a. E (X T ε) = 0
E (εεT ) = σ 2 I ), the OLS estimator is:
unbiased: E β̂ = β
consistent: β̂ converges to β with growing n
efficient: least possible estimator variance (i.e. highest
precision)
Measuring precision
n
1
1. Variance of the error term (scalar): σ̂ 2 = ε2i
P
n−(k+1)
i=1
Why such a formula if the general formula is
n
1 P
Var (X ) = n−1 (xi − x̄)2 ?
i=1
First of all note that ε̄ = 0 (prove it on your own).
Second, we need to know why 1 turned into (k + 1) in the
denominator.
Measuring precision
Without r
a correction in denominator:
√ h
2 2 2
i q
Var = 3 (3 − 2) + (2 − 2) + (1 − 2) = 23 6= 1
1
Measuring precision
Without r
a correction in denominator:
√ h
2 2 2
i q
Var = 3 (3 − 2) + (2 − 2) + (1 − 2) = 23 6= 1
1
Measuring precision
Measuring precision
Measuring precision
Measuring precision
T
Var β̂ = E β̂ − β β̂ − β =
h i h iT
−1 T −1 T
XT X X y − β · XT X
= E X y −β =
h i h iT
T
−1 T T
−1 T
= E X X X (Xβ + ε) − β · X X X (Xβ + ε) − β =
h −1 T −1 i
T T T
= E X X X ε·ε X X X =
T
−1 T T
T
−1
= X X X E εε X X X =
−1 T 2 −1
= XT X X σ IX XT X =
−1 T −1
= σ 2 XT X X X XT X =
2 T
−1
= σ X X
−1
Empirical counterpart: Var β̂ = σ̂ 2 XT X ≡ [di,j ](k+1)×(k+1)
Measuring precision
Standard
p p (vector
errors of estimation –
for each parameter):
p
s β̂0 = d1,1 s β̂1 = d2,2 s β̂2 = d3,3 . . .
Calculating S.E.
1. estimate parameters, 2. compute the empirical error terms, 3.
estimate their variance, 4. compute the variance-covariance matrix
of the OLS estimator, 5. compute the SE as a square root of its
diagonal elements.
Measuring precision
Standard
p p (vector
errors of estimation –
for each parameter):
p
s β̂0 = d1,1 s β̂1 = d2,2 s β̂2 = d3,3 . . .
Calculating S.E.
1. estimate parameters, 2. compute the empirical error terms, 3.
estimate their variance, 4. compute the variance-covariance matrix
of the OLS estimator, 5. compute the SE as a square root of its
diagonal elements.
Measuring precision
t-Student test
H0 : βi = 0, i.e. i-th explanatory variable does not significantly
influence y
H1 : βi 6= 0, i.e. i-th explanatory variable does not significantly
influence y
Test statistic: t = s β̂β̂i is distributed as t (n − k − 1).
( 1)
p-value<α∗ – reject H0
p-value>α∗ – do not reject H0
Measuring precision
Example (3/4)
R-squared (1)
R-squared (2)
R-squared (3)
R-squared (4)
R-squared (5)
R-squared (6)
R-squared (7)
Wald test
H0 : β1 = β2 = . . . = βk = 0, i.e. no explanatory variable
influences y
H1 : ∃i βi 6= 0, at least 1 explanatory variable influences y
2 /k
Test statistic: F = (1−R 2 R)/(T −k−1)
distributed as F (k, T − k − 1).
Adjusted R-squared
k
R¯2 = |{z}
R2 − 1 − R2
T − (k + 1)
fit | {z }
penalty for overparametrization
Example (4/4)