Simple Linear Regression

Week 5: Simple Linear Regression
Brandon Stewart1
Princeton
October 10, 12, 2016
1
These slides are heavily influenced by Matt Blackwell, Adam Glynn and Jens
Hainmueller. Illustrations by Shay O’Brien.
Stewart (Princeton) Week 5: Simple Linear Regression October 10, 12, 2016 1 / 103
Where We’ve Been and Where We’re Going...
Last Week
I hypothesis testing
I what is regression
This Week
I Monday:
F mechanics of OLS
F properties of OLS
I Wednesday:
F hypothesis tests for regression
F confidence intervals for regression
F goodness of fit
Next Week
I mechanics with two regressors
I omitted variables, multicollinearity
Long Run
I probability → inference → regression
Questions?
Macrostructure
The next few weeks,

Linear Regression with Two Regressors
Multiple Linear Regression
Break Week
Regression in the Social Science
What Can Go Wrong and How to Fix It Week 1
What Can Go Wrong and How to Fix It Week 2 / Thanksgiving
Causality with Measured Confounding
Unmeasured Confounding and Instrumental Variables
Repeated Observations and Panel Data
A brief comment on exams, midterm week etc.
1 Mechanics of OLS
2 Properties of the OLS estimator
3 Example and Review
4 Properties Continued
5 Hypothesis tests for regression
6 Confidence intervals for regression
7 Goodness of fit
8 Wrap Up of Univariate Regression
9 Fun with Non-Linearities
The population linear regression function
The (population) simple linear regression model can be stated as the

following:
r (x) = E [Y |X = x] = β0 + β1 x
This (partially) describes the data generating process in the
population
Y = dependent variable
X = independent variable
β0 , β1 = population intercept and population slope (what we want to
estimate)
The sample linear regression function
The estimated or sample regression function is:
rb(Xi ) = Ybi = βb0 + βb1 Xi
βb0 , βb1 are the estimated intercept and slope

Ybi is the fitted/predicted value
We also have the residuals, ubi which are the differences between the
true values of Y and the predicted value:
ubi = Yi − Ybi
You can think of the residuals as the prediction errors of our

estimates.
Overall Goals for the Week
Learn how to run and read regression

Mechanics: how to estimate the intercept and slope?
Properties: when are these good estimates?
Uncertainty: how will the OLS estimator behave in repeated samples?
Testing: can we assess the plausibility of no relationship (β1 = 0)?
Interpretation: how do we interpret our estimates?
What is OLS?
An estimator for the slope and the intercept of the regression line
We talked last week about ways to derive this estimator and we
settled on deriving it by minimizing the squared prediction errors of
the regression, or in other words, minimizing the sum of the squared
residuals:
Ordinary Least Squares (OLS):
n
X
(βb0 , βb1 ) = arg min (Yi − b0 − b1 Xi )2
b0 ,b1 i=1
In words, the OLS estimates are the intercept and slope that minimize
the sum of the squared residuals.
Graphical Example
How do we fit the regression line Ŷ = β̂0 + β̂1 X to the data?
 
 0 1
Graphical Example
How do we fit the regression line Ŷ = β̂0 + β̂1 X to the data?
Answer: We will minimize the squared sum of residuals
Residual ui is “part”
of Yi not predicted
 
ui  Yi  Y i
n  2
min
 
u i 1
i
0, 1
Deriving the OLS estimator
Let’s think about n pairs of sample observations:
(Y1 , X1 ), (Y2 , X2 ), . . . , (Yn , Xn )
Let {b0 , b1 } be possible values for {β0 , β1 }
Define the least squares objective function:
n
X
S(b0 , b1 ) = (Yi − b0 − b1 Xi )2 .
i=1
How do we derive the LS estimators for β0 and β1 ? We want to

minimize this function, which is actually a very well-defined calculus
problem.
1 Take partial derivatives of S with respect to b0 and b1 .
2 Set each of the partial derivatives to 0
3 Solve for {b0 , b1 } and replace them with the solutions
To the board we go!
The OLS estimator
Now we’re done! Here are the OLS estimators:
βb0 = Y − βb1 X
Pn
(Xi − X )(Yi − Y )
β1 = i=1
b Pn 2
i=1 (Xi − X )
Intuition of the OLS estimator
The intercept equation tells us that the regression line goes through
the point (Y , X ):
Y = βb0 + βb1 X
The slope for the regression line can be written as the following:
Pn
i=1 (Xi − X )(Yi − Y) Sample Covariance between X and Y
βb1 = Pn 2
=
i=1 (Xi − X )
Sample Variance of X
The higher the covariance between X and Y , the higher the slope will
be.
Negative covariances → negative slopes;
positive covariances → positive slopes
What happens when Xi doesn’t vary?
What happens when Yi doesn’t vary?
A Visual Intuition for the OLS Estimator
+
+
+ + -
+ +
+ + +
Mechanical properties of OLS
Later we’ll see that under certain assumptions, OLS will have nice
statistical properties.
But some properties are mechanical since they can be derived from
the first order conditions of OLS.
1 The residuals will be 0 on average:
n
1X
ubi = 0
n
i=1
2 The residuals will be uncorrelated with the predictor
(cov
c is the sample covariance):
cov(X
c i , ubi ) = 0
3 The residuals will be uncorrelated with the fitted values:
cov(
c Ybi , ubi ) = 0
OLS slope as a weighted sum of the outcomes
One useful derivation is to write the OLS estimator for the slope as a
weighted sum of the outcomes.
n
X
βb1 = Wi Yi
i=1
Where here we have the weights, Wi as:
(Xi − X )
Wi = Pn 2
i=1 (Xi − X )
This is important for two reasons. First, it’ll make derivations later
much easier. And second, it shows that is just the sum of a random
variable. Therefore it is also a random variable.
To the board!
1 Mechanics of OLS
7 Goodness of fit
Sampling distribution of the OLS estimator
Remember: OLS is an estimator—it’s a machine that we plug data
into and we get out estimates.
Sample 1: {(Y1 , X1 ), . . . , (Yn , Xn )} (βb0 , βb1 )1
Sample 2: {(Y1 , X1 ), . . . , (Yn , Xn )} (βb0 , βb1 )2
.. OLS ..
. .
Sample k − 1: {(Y1 , X1 ), . . . , (Yn , Xn )} (βb0 , βb1 )k−1
Sample k: {(Y1 , X1 ), . . . , (Yn , Xn )} (βb0 , βb1 )k
Just like the sample mean, sample difference in means, or the sample
variance
It has a sampling distribution, with a sampling variance/standard
error, etc.
Let’s take a simulation approach to demonstrate:
I Pretend that the AJR data represents the population of interest
I See how the line varies from sample to sample
Simulation procedure
1 Draw a random sample of size n = 30 with replacement using

sample()
2 Use lm() to calculate the OLS estimates of the slope and intercept
3 Plot the estimated regression line
Population Regression
12
Log GDP per capita growth
11
10
9
8
7
6
1 2 3 4 5 6 7 8
Log Settler Mortality
Randomly sample from AJR
12
11
10
9
8
7
6
1 2 3 4 5 6 7 8
Sampling distribution of OLS
You can see that the estimated slopes and intercepts vary from sample
to sample, but that the “average” of the lines looks about right.
Sampling distribution of intercepts Sampling distribution of slopes
300
Frequency
Frequency
300
100
100
0
0
6 8 10 12 14 -1.5 -1.0 -0.5 0.0 0.5
^
β0 ^
β1
Is this unique?
Assumptions for unbiasedness of the sample mean
What assumptions did we make to prove that the sample mean was
unbiased?
E[X ] = µ
Just one: random sample
We’ll need more than this for the regression case
Our goal
What is the sampling distribution of the OLS slope?
βb1 ∼?(?, ?)
We need fill in those ?s.

We’ll start with the mean of the sampling distribution. Is the
estimator centered at the true value, β1 ?
Most of our derivations will be in terms of the slope but they apply to
the intercept as well.
OLS Assumptions Preview
1 Linearity in Parameters: The population model is linear in its

parameters and correctly specified
2 Random Sampling: The observed data represent a random sample
from the population described by the model.
3 Variation in X : There is variation in the explanatory variable.
4 Zero conditional mean: Expected value of the error term is zero
conditional on all values of the explanatory variable
5 Homoskedasticity: The error term has the same variance conditional
on all values of the explanatory variable.
6 Normality: The error term is independent of the explanatory variables
and normally distributed.
Hierarchy of OLS Assumptions
@(A--B?(.C)D*EF<3GH* 5:(--/'(:*<?*EF3GH*
!"#$%&'(%)$* 3$4/(-#"$#--*
I-680,)%'*!$J#.#$'#************* 98(::B9(80:#*!$J#.#$'#***
+(,(*+#-'./0%)$* 5)$-/-,#$'6* EK*($"*!LH" E,*($"*NH*
1(./(%)$*/$*2* 1(./(%)$*/$*2* 1(./(%)$*/$*2* 1(./(%)$*/$*2*
7($")8*9(80:/$;* 7($")8*9(80:/$;* 7($")8*9(80:/$;*
</$#(./,6*/$* </$#(./,6*/$* </$#(./,6*/$*

=(.(8#,#.-* =(.(8#,#.-* =(.(8#,#.-*
>#.)*5)$"/%)$(:* >#.)*5)$"/%)$(:* >#.)*5)$"/%)$(:*

?#($* ?#($* ?#($*
M)8)-C#"(-%'/,6* M)8)-C#"(-%'/,6*
O).8(:/,6*)J*G..).-*
OLS Assumption I
Assumption (I. Linearity in Parameters)
The population regression model is linear in its parameters and correctly
specified as:
Y = β0 + β1 X1 + u
Note that it can be nonlinear in variables

I OK: Y = β0 + β1 X + u or
Y = β0 + β1 X 2 + u or
Y = β0 + β1 log (X ) + u
I Not OK: Y = β0 + β12 X + u or
Y = β0 + exp(β1 )X + u
β0 , β1 : Population parameters — fixed and unknown

u: Unobserved random variable with E [u] = 0 — captures all other
factors influencing Y other than X
We assume this to be the structural model, i.e., the model describing
the true process generating Y
OLS Assumption II
Assumption (II. Random Sampling)

The observed data:
(yi , xi ) for i = 1, ..., n
represent an i.i.d. random sample of size n following the population model.
Data examples consistent with this assumption:

A cross-sectional survey where the units are sampled randomly
Potential Violations:
Time series data (regressor values may exhibit persistence)
Sample selection problems (sample not representative of the
population)
OLS Assumption III
Assumption (III. Variation in X ; a.k.a. No Perfect Collinearity)
The observed data:
xi for i = 1, ..., n
are not all the same value.
Satisfied as long as there is some variation in the regressor X in the
sample.
Why do we need this?

Pn
(x − x̄)(yi − ȳ )
β̂1 = Pn i
i=1
2
i=1 (xi − x̄)
This assumption is needed just to calculate β̂, i.e. identifying β̂.

In fact, this is the only assumption needed for using OLS as a pure data
summary.
Stuck in a moment
Why does this matter? How would you draw the line of best fit
through this scatterplot, which is a violation of this assumption?
1
0
Y
-1
-2
-3 -2 -1 0 1 2 3
X
OLS Assumption IV
Assumption (IV. Zero Conditional Mean)

The expected value of the error term is zero conditional on any value of the
explanatory variable:
E [u|X ] = 0
E [u|X ] = 0 implies a slightly weaker condition Cov(X , u) = 0

Given random sampling, E [u|X ] = 0 also implies E [ui |xi ] = 0 for all i
Violations:
Recall that u represents all unobserved factors that influence Y
If such unobserved factors are also correlated with X , Cov(X , u) 6= 0
Example: Wage = β0 + β1 education + u. What is likely to be in u?

−→ It must be assumed E [ability |educ = low ] = E [ability |educ = high]
Violating the zero conditional mean assumption
How does this assumption get violated? Let’s generate data from the
following model:
Yi = 1 + 0.5Xi + ui
But let’s compare two situations:

1 Where the mean of ui depends on Xi (they are correlated)
2 No relationship between them (satisfies the assumption)
Violating the zero conditional mean assumption
Assumption 4 violated Assumption 4 not violated

5
5
4
4
3
3
2
2
Y
Y
1
1
0
0
-1
-1
-2
-2
-3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3
X X
Unbiasedness (to the blackboard)
With Assumptions 1-4, we can show that the OLS estimator for the slope
is unbiased, that is E [βb1 ] = β1 .
Unbiasedness of OLS
Theorem (Unbiasedness of OLS)

Given OLS Assumptions I–IV:
E [β̂0 ] = β0 and E [β̂1 ] = β1
The sampling distributions of the estimators β̂1 and β̂0 are centered about
the true population parameter values β1 and β0 .
Where are we?
Now we know that, under Assumptions 1-4, we know that
βb1 ∼?(β1 , ?)
That is we know that the sampling distribution is centered on the

true population slope, but we don’t know the population variance.
Sampling variance of estimated slope
In order to derive the sampling variance of the OLS estimator,
1 Linearity
2 Random (iid) sample
3 Variation in Xi
4 Zero conditional mean of the errors
5 Homoskedasticity
Variance of OLS Estimators
How can we derive Var[β̂0 ] and Var[β̂1 ]? Let’s make the following additional
assumption:
Assumption (V. Homoskedasticity)

The conditional variance of the error term is constant and does not vary as a
function of the explanatory variable:
Var[u|X ] = σu2
This implies Var[u] = σu2

→ all errors have an identical error variance (σu2i = σu2 for all i)
Taken together, Assumptions I–V imply:
E [Y |X ] = β0 + β1 X
Var[Y |X ] = σu2
Violation: Var[u|X = x1 ] 6= Var[u|X = x2 ] called heteroskedasticity.

Assumptions I–V are collectively known as the Gauss-Markov assumptions
Deriving the sampling variance
var[βb1 |X1 , . . . , Xn ] =??
Variance of OLS Estimators
Theorem (Variance of OLS Estimators)
Given OLS Assumptions I–V (Gauss-Markov Assumptions):
σu2 σu2
Var[β̂1 | X ] = Pn 2
=
i=1 (xi − x̄) SSTx
x̄ 2

2 1
Var[β̂0 | X ] = σu + Pn 2
n i=1 (xi − x̄)
where Var[u | X ] = σu2 (the error variance).
Understanding the sampling variance
σu2
var[βb1 |X1 , . . . , Xn ] = Pn 2
i=1 (Xi − X )
What drives the sampling variability of the OLS estimator?

I The higher the variance of Yi , the higher the sampling variance
I The lower the variance of Xi , the higher the sampling variance
I As we increase n, the denominator gets large, while the numerator is
fixed and so the sampling variance shrinks to 0.
Estimating the Variance of OLS Estimators
How can we estimate the unobserved error variance Var [u] = σu2 ?
We can derive an estimator based on the residuals:
ûi = yi − ŷi = yi − β̂0 − β̂1 xi
Recall: The errors ui are NOT the same as the residuals ûi .
Intuitively, the scatter of the residuals around the fitted regression line should
reflect the unseen scatter about the true population regression line.
We can measure scatter with the mean squared deviation:
n n
1X ¯ 2= 1
X
MSD(û) ≡ (ûi − û) ûi2
n n
i=1 i=1
Intuitively, which line is likely to be closer to the observed sample values on X

and Y , the true line yi = β0 + β1 xi or the fitted regression line ŷi = β̂0 + β̂1 xi ?
Estimating the Variance of OLS Estimators
By construction, the regression line is closer since it is drawn to fit the
actual sample we have
Specifically, the regression line is drawn so as to minimize the sum of the
squares of the distances between it and the observations
So the spread of the residuals MSD(û) will slightly underestimate the error
variance Var[u] = σu2 on average
In fact, we can show that with a single regressor X we have:
n−2 2
E [MSD(û)] = σu (degrees of freedom adjustment)
n
Thus, an unbiased estimator for the error variance is:

n n
n n 1X 1 X 2
σ̂u2 = MSD(û) = ûi = ûi
n−2 n−2n n−2
i=1 i=1
We plug this estimate into the variance estimators for β̂0 and β̂1 .
Where are we?
Under Assumptions 1-5, we know that
σu2

β1 ∼? β1 , Pn
b
2
i=1 (Xi − X )
Now we know the mean and sampling variance of the sampling

distribution.
Next Time: how does this compare to other estimators for the
population slope?
Where We’ve Been and Where We’re Going...
Last Week
I hypothesis testing
I what is regression
This Week
I Monday:
F mechanics of OLS
F properties of OLS
I Wednesday:
F hypothesis tests for regression
F confidence intervals for regression
F goodness of fit
Next Week
I mechanics with two regressors
I omitted variables, multicollinearity
Long Run
I probability → inference → regression
Questions?
1 Mechanics of OLS
7 Goodness of fit
Example: Epstein and Mershon SCOTUS data
Data on 27 justices from the Warren, Burger, and Rehnquist courts

(can be interpreted as a census)
Percentage of votes in liberal direction for each justice in a number of
issue areas
Segal-Cover scores for each justice
Party of appointing president
Douglas
90
Goldberg
Marshall
Fortas
80 Warren Brennan
Black
y = 27.6 + 41.2x + u
70
Stevens ____
Rise
60
Run
CLlib
Blackmun Stewart
50
Frankfurter
Jackson
Clark
Whittaker Harlan
Souter White
40
Burton
Powell Minton
Kennedy
O'Connor Reed
30
Scalia Burger
●
Thomas
Rehnquist
20
0.0 0.2 0.4 0.6 0.8 1.0
SCscore
How to get β0 and β1
β̂0 = ȳ − β̂1 x̄.
Pn
(x − x̄)(yi − ȳ )
β̂1 = Pn i
i=1
2
.
i=1 (xi − x̄)
1 Mechanics of OLS
7 Goodness of fit
Where are we?
@(A--B?(.C)D*EF<3GH* 5:(--/'(:*<?*EF3GH*
!"#$%&'(%)$* 3$4/(-#"$#--*
I-680,)%'*!$J#.#$'#************* 98(::B9(80:#*!$J#.#$'#***
+(,(*+#-'./0%)$* 5)$-/-,#$'6* EK*($"*!LH" E,*($"*NH*
1(./(%)$*/$*2* 1(./(%)$*/$*2* 1(./(%)$*/$*2* 1(./(%)$*/$*2*
7($")8*9(80:/$;* 7($")8*9(80:/$;* 7($")8*9(80:/$;*
</$#(./,6*/$* </$#(./,6*/$* </$#(./,6*/$*

=(.(8#,#.-* =(.(8#,#.-* =(.(8#,#.-*
>#.)*5)$"/%)$(:* >#.)*5)$"/%)$(:* >#.)*5)$"/%)$(:*

?#($* ?#($* ?#($*
M)8)-C#"(-%'/,6* M)8)-C#"(-%'/,6*
O).8(:/,6*)J*G..).-*
Where are we?
σu2

βb1 ∼? β1 , Pn 2
i=1 (Xi − X )
Now we know the mean and sampling variance of the sampling

distribution.
How does this compare to other estimators for the population slope?
OLS is BLUE :(
Theorem (Gauss-Markov)
Given OLS Assumptions I–V, the OLS estimator is BLUE, i.e. the
1 Best: Lowest variance in class
2 Linear: Among Linear estimators
3 Unbiased: Among Linear Unbiased estimators
4 Estimator.
Assumptions 1-5: the “Gauss Markov Assumptions”

The proof is detailed and doesn’t yield insight, so we skip it. (We will
explore the intuition some more in a few slides)
Fails to hold when the assumptions are violated!
Gauss-Markov Theorem
OLS is efficient in the class of unbiased, linear estimators.
All estimators
unbiased
linear
OLS is BLUE--best linear unbiased estimator.

Where are we?
σu2

β1 ∼? β1 , Pn
b
2
i=1 (Xi − X )
σu2
And we know that Pn (X 2 is the lowest variance of any linear
i=1 i −X )
estimator of β1
What about the last question mark? What’s the form of the
distribution? Uniform? t? Normal? Exponential? Hypergeometric?
Large-sample distribution of OLS estimators
Remember that the OLS estimator is the sum of independent r.v.’s:
n
X
βb1 = Wi Yi
i=1
Mantra of the Central Limit Theorem:

“the sums and means of r.v.’s tend to be Normally distributed in
large samples.”
True here as well, so we know that in large samples:

βb1 − β1
∼ N(0, 1)
SE [βb1 ]
Can also replace SE with an estimate:
βb1 − β1
∼ N(0, 1)
SE
c [βb1 ]
Where are we?
Under Assumptions 1-5 and in large samples, we know that
σu2

β1 ∼ N β1 , Pn
b
2
i=1 (Xi − X )
Sampling distribution in small samples
What if we have a small sample? What can we do then?

Can’t get something for nothing, but we can make progress if we
make another assumption:
1 Linearity
2 Random (iid) sample
3 Variation in Xi
4 Zero conditional mean of the errors
5 Homoskedasticity
6 Errors are conditionally Normal
OLS Assumptions VI
Assumption (VI. Normality)
⊥X , and
The population error term is independent of the explanatory variable, u⊥
is normally distributed with mean zero and variance σu2 :
u ∼ N(0, σu2 ), which implies Y |X ∼ N(β0 + β1 X , σu2 )
Note: This implies homoskedasticity and zero conditional mean.
Together Assumptions I–VI are the classical linear model (CLM)

assumptions.
The CLM assumptions imply that OLS is BUE (i.e. minimum variance
among all linear or non-linear unbiased estimators)
Non-normality of the errors is a serious concern in small samples. We can
partially check this assumption by looking at the residuals
Variable transformations can help to come closer to normality
We don’t need normality assumption in large samples
Sampling Distribution for βb1
Theorem (Sampling Distribution of βb1 )
Under Assumptions I–VI,

βb1 ∼ N β1 , Var[βb1 | X ]
where σu2
Var[β̂1 | X ] = Pn 2
i=1 (xi − x̄)
which implies βb − β1 βb1 − β1
q 1 = ∼ N(0, 1)
Var[β̂1 | X ] SE (β̂)
Proof.
Given Assumptions I–VI, β̂1 is a linear combination of the i.i.d. normal random variables:
n
X (xi − x̄)
β̂1 = β1 + ui where ui ∼ N(0, σu2 ).
i=1
SSTx
Any linear combination of independent normals is normal, and we can transform/standarize any
normal random variable into a standard normal by subtracting off its mean and dividing by its
standard deviation.
Sampling distribution of OLS slope
If we have Yi given Xi is distributed N(β0 + β1 Xi , σu2 ), then we have
the following at any sample size:
βb1 − β1
∼ N(0, 1)
SE [βb1 ]
Furthermore, if we replace the true standard error with the estimated

standard error, then we get the following:
βb1 − β1
∼ tn−2
SE
c [βb1 ]
The standardized coefficient follows a t distribution n − 2 degrees of

freedom. We take off an extra degree of freedom because we had to
one more parameter than just the sample mean.
All of this depends on Normal errors! We can check to see if the error
do look Normal.
The t-Test for Single Population Parameters
SE [β̂1 ] = √Pn σu 2
involves the unknown population error variance σu2
i=1 (xi −x̄)
Pn 2
i=1 ûi
Replace σu2 with its unbiased estimator σ̂u2 = n−2 , and we obtain:
Theorem (Sampling Distribution of t-value)

Under Assumptions I–VI, the t-value for β1 has a t-distribution with n − 2 degrees
of freedom:
βb1 − β1
T ≡ ∼ τn−2
\
SE [β̂1 ]
Proof.
The logic is perfectly analogous to the t-value for the population mean — because we
are estimating the denominator, we need a distribution that has fatter tails than N(0, 1)
to take into account the additional uncertainty.
This time, σ̂u2 contains two estimated parameters (β̂0 and β̂1 ) instead of one, hence the
degrees of freedom = n − 2.
Where are we?
Under Assumptions 1-5 and in large samples, we know that
σu2

βb1 ∼ N β1 , Pn 2
i=1 (Xi − X )
Under Assumptions 1-6 and in any sample, we know that
βb1 − β1
∼ tn−2
SE
c [βb1 ]
Now let’s briefly return to some of the large sample properties.
Large Sample Properties: Consistency
We just looked formally at the small sample properties of the OLS
estimator, i.e., how (β̂0 , β̂1 ) behaves in repeated samples of a given n.
Now let’s take a more rigorous look at the large sample properties, i.e., how
(β̂0 , β̂1 ) behaves when n → ∞.
Theorem (Consistency of OLS Estimator)

Given Assumptions I–IV, the OLS estimator βb1 is consistent for β1 as n → ∞:
plim βb1 = β1
n→∞
Technical note: We can slightly relax Assumption IV:

E [u|X ] = 0 (any function of X is uncorrelated with u)
to its implication:
Cov[u, X ] = 0 (X is uncorrelated with u)
for consistency to hold (but not unbiasedness).
Proof.
Similar to the unbiasedness proof:
Pn Pn
(xi − x̄)yi (xi − x̄)ui
β̂1 = Pi=1 n = β 1 + Pi n
i=1 (xi − x̄)
2
i (xi − x̄)
2
Pn
(xi − x̄)ui
plim βb1 = plim β1 + plim Pi n (Wooldridge C.3 Property i)
i (xi − x̄)
2
plim n1 ni (xi − x̄)ui

P
= β1 + (Wooldridge C.3 Property iii)
plim n1 ni (xi − x̄)2
P
Cov[X , u]
= β1 + (by the law of large numbers)
Var[X ]
= β1 (Cov[X , u] = 0 and Var[X ] > 0)
OLS is inconsistent (and biased) unless Cov[X , u] = 0

If Cov[u, X ] > 0 then asymptotic bias is upward; if Cov[u, X ] < 0
asymptotic bias is downwards
FIGURE 5.1

Sampling distributions of (3, for sample sizes n, < n2 < n3 .
n3
{p,
n2
-1 _
Il, ~,
Sampling distributions of β̂1 , for sample sizes n1 < n2 < n3
Large Sample Properties: Asymptotic Normality
For statistical inference, we need to know the sampling distribution of β̂
when n → ∞.
Theorem (Asymptotic Normality of OLS Estimator)

Given Assumptions I–V, the OLS estimator βb1 is asymptotically normally
distributed:
β̂1 − β1 approx.
∼ N(0, 1)
SE
c [β̂1 ]
where
σ̂u
SE
c [β̂1 ] = q
Pn 2
i=1 (xi − x̄)
with the consistent estimator for the error variance:

n
1X 2 p 2
σ̂u2 = ûi → σu
n
i=1
Large Sample Inference
Proof.
Proof is similar to the small-sample normality proof:
n
X (xi − x̄)
β̂1 = β1 + ui
i=1
SSTx
√ 1 Pn
√ n · n i=1 (xi − x̄)ui
n(β̂1 − β1 ) = 1
Pn
i=1 (xi − x̄)
2
n
where the numerator converges in distribution to a normal random variable by CLT.
Then, rearranging the terms, etc. gives you the right formula given in the theorem.
For a more formal and detailed proof, see Wooldridge Appendix 5A.
We need homoskedasticity (Assumption V) for this result, but we do not need

normality (Assumption VI).
Result implies that asymptotically our usual standard errors, t-values, p-values, and
CIs remain valid even without the normality assumption! We just proceed as in the
small sample case where we assume normality.
It turns out that, given Assumptions I–V, the OLS asymptotic variance is also the
lowest in class (asymptotic Gauss-Markov).
Testing and Confidence Intervals
Three ways of making statistical inference out of regression:

1 Point Estimation: Consider the sampling distribution of our point
estimator β̂1 to infer β1
2 Hypothesis Testing: Consider the sampling distribution of a test
statistic to test hypothesis about β1 at the α level
3 Interval Estimation: Consider the sampling distribution of an interval
estimator to construct intervals that will contain β1 at least
100(1 − α)% of the time.
For 2 and 3, we need to know more than just the mean and the variance of
the sampling distribution of β̂1 . We need to know the full shape of the
sampling distribution of our estimators β̂0 and β̂1 .
1 Mechanics of OLS
7 Goodness of fit
Null and alternative hypotheses review
Null: H0 : β1 = 0
I The null is the straw man we want to knock down.
I With regression, almost always null of no relationship
Alternative: Ha : β1 6= 0
I Claim we want to test
I Almost always “some effect”
I Could do one-sided test, but you shouldn’t
Notice these are statements about the population parameters, not the
OLS estimates.
Test statistic
Under the null of H0 : β1 = c, we can use the following familiar test
statistic:
βb1 − c
T =
SE
c [βb1 ]
As we saw in the last section, if the errors are conditionally Normal,
then under the null hypothesis we have:
T ∼ tn−2
In large samples, we know that T is approximately (standard)

Normal, but we also know that tn−2 is approximately (standard)
Normal in large samples too, so this statement works there too, even
if Normality of the errors fails.
Thus, under the null, we know the distribution of T and can use that
to formulate a rejection region and calculate p-values.
Rejection region
Choose a level of the test, α, and find rejection regions that
correspond to that value under the null distribution:
P(−tα/2,n−2 < T < tα/2,n−2 ) = 1 − α
This is exactly the same as with sample means and sample differences
in means, except that the degrees of freedom on the t distribution
have changed.
0.5
Reject Retain Reject
0.4
0.3
dnorm(x)
0.2
0.1
0.025 0.025
0.0
-t = -1.96 t = 1.96
-4 -2 0 2 4
x
p-value
The interpretation of the p-value is the same: the probability of seeing

a test statistic at least this extreme if the null hypothesis were true
Mathematically: !
βb − c
1
≥ |Tobs |

P
SE
c [βb1 ]
If the p-value is less than α we would reject the null at the α level.
1 Mechanics of OLS
7 Goodness of fit
Confidence intervals
Very similar to the approach with sample means. By the sampling
distribution of the OLS estimator, we know that we can find t-values
such that:
βb1 − β1
P − tα/2,n−2 ≤ ≤ tα/2,n−2 = 1 − α
SE
c [βb1 ]
If we rearrange this as before, we can get an expression for confidence
intervals:

P βb1 − tα/2,n−2 SE c [βb1 ] = 1 − α
c [βb1 ] ≤ β1 ≤ βb1 + tα/2,n−2 SE
Thus, we can write the confidence intervals as:

βb1 ± tα/2,n−2 SE
c [βb1 ]
We can derive these for the intercept as well:

βb0 ± tα/2,n−2 SE
c [βb0 ]
Sampling distribution of interval estimates
CIs Simulation Example
Returning to the simulation example, we can simulate the sampling distributions of the
95% interval estimates for βb0 and βb1 .
Returning to our simulation example we can simulate the sampling
distributions of the 95 % confidence interval estimates for βb1 and βb0
●
6
●
● ●
4
●
●
●
●
2
●
●
●
yy
●
●
−2
●
●
●
●
−4
●
●●
●
−6
●
●
0 2 4 6 8 10 0 2 4 6 8 10 −2.0 −1.5 −1.0 −0.5 0.0

^ ^
xx β0 β1
Gov2000: Quantitative Methodology for Political Science I
When we repeat the process over and over, we expect 95% of the confidence intervals
CIs Simulation
to contain Example
the true parameters.
Note that, in a given sample, one CI may cover its true value and the other may not.
0 2 4 6 8 10 −2.0 −1.5 −1.0 −0.5 0.0

^ ^
β0 β1
Gov2000: Quantitative Methodology for Political Science I
Prediction error
How do we judge how well a line fits the data?
One way is to find out how much better we do at predicting Y once
we include X into the regression model.
Prediction errors without X : best prediction is the mean, so our
squared errors, or the total sum of squares (SStot ) would be:
n
X
SStot = (Yi − Y )2
i=1
Once we have estimated our model, we have new prediction errors,

which are just the sum of the squared residuals or SSres :
n
X
SSres = (Yi − Ybi )2
i=1
Sum of Squares
Total Prediction Errors

12
11
10
9
8
7
6
1 2 3 4 5 6 7 8
Sum of Squares
Residuals
12
11
10
9
8
7
6
1 2 3 4 5 6 7 8
R-square
By definition, the residuals have to be smaller than the deviations

from the mean, so we might ask the following: how much lower is the
SSres compared to the SStot ?
We quantify this question with the coefficient of determination or R 2 .
This is the following:
SStot − SSres SSres
R2 = =1−
SStot SStot
This is the fraction of the total prediction error eliminated by
providing information on X .
Alternatively, this is the fraction of the variation in Y is “explained
by” X .
R 2 = 0 means no relationship
R 2 = 1 implies perfect linear fit
Is R-squared useful?
15 ●
●
● ●
●
●
● ●
●●
10
●
●● ●
●
● ●
● ●
● ●
●
●
● ● ● ● ●
● ● ●
● ●
● ●
●
y
● ●
● ● ●
5
● ● ●
●● ● ● ● ●
●
●● ●● ●
● ●
●● ● ●
●
● ●●
● ● ●
● ●
● ●
●
● ●
●
●
●
●
● R−squared = 0.66
●
● ●
●
● ●
0
●
●● ●
● ● ●
●
●
●
●
● ●
Stewart (Princeton)0 2 Week 5: Simple

4 Linear Regression
6 8 October 10,
10 12, 2016 80 / 103
●
●
● ●●
10
●●
● ● ●
●
● ●● ●
●
●
●
8
●
● ●● ● ●
●●
●●
● ●
●
● ●
6
●
● ●
●
●
y
●
●
● ● ● ● ●
●●
● ● ● ● ●
4
●
● ●
● ● ● ●
● ● ● ●●
●
●
●
●● ● ● ●
●
●●
●●
R−squared = 0.96
2
●
● ● ●
●
●● ●
●● ● ●
●
●
● ●
●●
● ●
●●
0
Stewart (Princeton)0 2 Week 5: Simple

4 Linear Regression
6 8 October 10,
10 12, 2016 80 / 103
12
12
10
10
Y
Y
8
8
6
6
4
4
5 10 15 5 10 15
X X
12
12
10
10
Y
Y
8
8
6
6
4
5 10 15 5 10 15
X X
Why r 2 ?
To calculate r 2 , we need to think about the following two quantities:
1 TSS: Total sum of squares
2 SSE: Sum of squared errors
n
X
TSS = (yi − ȳ )2 .
i=1
n
X
SSE = ui2 .
i=1
SSE
r2 = 1 − .
TSS
Douglas
TSS
90
Goldberg
Marshall
Fortas
80
Warren Brennan
Black
70
Stevens
60
CLlib
Blackmun Stewart
50
Frankfurter
Jackson
Clark
Whittaker Harlan
Souter White
40
Burton
Powell Minton
Kennedy
O'Connor Reed
30
Scalia Burger
Thomas
Rehnquist
20
0.0 0.2 0.4 0.6 0.8 1.0
SCscore
Douglas
TSS
90
Goldberg
SSE
Marshall
Fortas
80
Warren Brennan
1−SSE/TSS = 0.45 Black

70
Stevens
60
CLlib
Blackmun Stewart
50
Frankfurter
Jackson
Clark
Whittaker Harlan
Souter White
40
Burton
Powell Minton
Kennedy
O'Connor Reed
30
Scalia Burger
Thomas
Rehnquist
20
0.0 0.2 0.4 0.6 0.8 1.0
SCscore
Derivation
n
X n
X
2
(yi − ȳ ) = yi − ȳ )}2
{ûi + (b
i=1 i=1
n
X
= {ûi2 + 2ûi (b yi − ȳ )2 }
yi − ȳ ) + (b
i=1
n
X n
X n
X
= ûi2 +2 yi − ȳ ) +
ûi (b yi − ȳ )2
(b
i=1 i=1 i=1
X n n
X
= ûi2 + yi − ȳ )2
(b
i=1 i=1
TSS = SSE + RegSS
Coefficient of Determination
We can divide each side by the TSS:

SSE RegSS TSS
+ =
TSS TSS TSS
SSE RegSS
+ =1
TSS TSS
RegSS SSE
=1− = r2
TSS TSS
r 2 is a measure of how much of the variation in Y is accounted for by X .
1 Mechanics of OLS
7 Goodness of fit
OLS Assumptions Summary
@(A--B?(.C)D*EF<3GH* 5:(--/'(:*<?*EF3GH*
!"#$%&'(%)$* 3$4/(-#"$#--*
I-680,)%'*!$J#.#$'#************* 98(::B9(80:#*!$J#.#$'#***
+(,(*+#-'./0%)$* 5)$-/-,#$'6* EK*($"*!LH" E,*($"*NH*
1(./(%)$*/$*2* 1(./(%)$*/$*2* 1(./(%)$*/$*2* 1(./(%)$*/$*2*
7($")8*9(80:/$;* 7($")8*9(80:/$;* 7($")8*9(80:/$;*
</$#(./,6*/$* </$#(./,6*/$* </$#(./,6*/$*

=(.(8#,#.-* =(.(8#,#.-* =(.(8#,#.-*
>#.)*5)$"/%)$(:* >#.)*5)$"/%)$(:* >#.)*5)$"/%)$(:*

?#($* ?#($* ?#($*
M)8)-C#"(-%'/,6* M)8)-C#"(-%'/,6*
O).8(:/,6*)J*G..).-*
What Do the Regression Coefficients Mean Substantively?
So far, we have learned the statistical properties of the OLS estimator
However, these properties do not tell us what types of inference we
can draw from the estimates
Three types of inference:

1 Descriptive inference:
I Summarizing sample data by drawing the “best fitting” line

I No inference about the underlying population intended
I Assumption required: III (variation in X ) only
2 Predictive inference:
I Inference about a new observation coming from the same population
I Example: Wage (Y ) and education (X ):
“What’s my best guess about the wage of a new worker who only has
high school education?”
I Assumptions required: III and II (random sampling)
I Assumptions desired: I (linearity)
What Do the Regression Coefficients Mean Substantively?
3 Causal inference:
I Inference about counterfactuals, i.e. hypothetical interventions to the
same units
I Example: Wage (Y ) and education (X ):
“What would my current wage be if I only had high school education?”
I Assumptions required (under the current framework): I, II, III and IV
(zero conditional mean)
I In this sequence we will continue to discuss causal identification
assumptions
Notice in the wage example, how the omission of unobserved ability

from the equation does or does not affect each type of inference
Implications:
I When Assumptions I–IV are all satisfied, we can estimate the structural
parameters β without bias and thus make causal inference.
I However, we can make predictive inference even if some assumptions
are violated.
OLS as a Best Linear Predictor (Review of BLUE)
Suppose that we want to predict the values of Y given observed X values
Suppose further that we’ve decided to use a linear predictor β̂0 + β̂1 X (but
not necessarily assume a true linear relationship in the population)
How to choose a good predictor? A popular criterion is mean squared error:

h i h i
MSE = E (Yi − Ŷi )2 = E (Yi − β̂0 − β̂1 Xi )2 = E ûi2

The smaller a predictor makes MSE , the better.

1
Pn
Now, note that the sample version of MSE = n i=1 ûi2
Pn
Recall how we got the OLS estimator; we minimized i=1 û 2 !
This implies that OLS is the best linear predictor in terms of MSE
Which assumptions did we use to get this result?

I Needed: Assumptions II (random sampling) and III (variation in X )
I Not needed: Assumptions I (linearity) and IV (zero cond. mean)
Note that Assumption I would make OLS the best, not just best linear,
predictor, so it is certainly desired
State Legislators and African American Population
Interpretations of increasing quality:
> summary(lm(beo ~ bpop, data = D))
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -1.31489 0.32775 -4.012 0.000264 ***
bpop 0.35848 0.02519 14.232 < 2e-16 ***
---
Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1
Residual standard error: 1.317 on 39 degrees of freedom

Multiple R-squared: 0.8385,Adjusted R-squared: 0.8344
F-statistic: 202.6 on 1 and 39 DF, p-value: < 2.2e-16
“A one percentage point increase in the African American population is associated with
a 0.35 percentage point increase in the fraction of African American state legislators
(p < 0.001).”
Ground Rules: Interpretation of the Slope
1 Give a short, but precise interpretation of the exact meaning of the

value of the slope coefficient referring to the concepts, units,
direction, and magnitude.
I Estimate suggests that one additional hour of reading the textbook is
associated with 10 additional points on the exam.
2 Do not resort to unwarranted causal language: Say “predicts”,

“associated with”, “expected difference” or “correlated with” instead
of “causes”, “leads” or “affects”
3 Give a short, but precise interpretation of statistical significance
4 Give a short, but precise interpretation of practical significance. You
want to discuss the magnitude of the slope in your particular
application.
Reporting Statistical Significance
A reasonable way to think about statistical significance is to think

about the precision of the estimates
If the slope is large substantively but just barely fails to reach

conventional levels of significance it may still be interesting.
Examples:
I We reject the null hypothesis that the slope is zero at the .05 level
I The slope coefficient suggests that a one unit change in X is
associated with a 10 unit change in Y (p.< .02).
I The slope coefficient is fairly precisely estimated, the 95 % confidence
interval ranging from 8 to 10
Reporting Substantive Significance
Statistical significance and substantive significance are not the same: with a
large enough sample size even truly microscopic differences can be
statistically significant!
To comment on substantive magnitude you should set up a “plausible”

contrast keeping in mind (1) the distributions of variables and the (2) the
substantive context
Examples:
Earnings on Schooling: The standard deviation is 2.5 years for schooling and
$50,000 for annual earnings. Thus, the slope estimates suggest that a one
standard deviation increase in schooling is associated with a .8 standard
deviation increase in earnings.
Next Week
OLS with two regressors

Omitted Variables and Multicolinearity
Dummy variables, interactions, polynomials
Reading:
I Fox Chapter 5.2.1 (Least Squares with Two Variables)
I Fox Chapter 7.1-7.3 (Dummy-Variable Regression, Interactions)
1 Mechanics of OLS
7 Goodness of fit
Fun with Non-Linearities
The linear regression model can accommodate non-linearity in X (but
not in β)
We do this by first transforming X appropriately
A useful transformation when variables are positive and right-skewed

is the (natural) logarithm
The log transformation changes the interpretation of β1 :

I Regress log(Y ) on X −→ β1 approximates percent increase in Y
associated with one unit increase in X
I Regress Y on log(X ) −→ β1 approximates increase in Y associated
with a percent increase in X
I Note that these approximations work only for small increments
I In particular, they do not work when X is a discrete random variable
Example from the American War Library
World War II
●
6e+05
Y: Numbers of American Soldiers Wounded in Action
5e+05
4e+05
3e+05
Civil War, North

●
2e+05
World War I
●
Vietnam War
● Civil War, South
●
1e+05
Korean War
●
Okinawa
Operation Iraqi Freedom,
●
Iraq
Iwo Jima
Revolutionary
●Republic War
0e+00
Operation Terrorism,
Terrorism
PersianTerrorism
Gulf,
Russia
Moro
China
Terrorism
OperationRussia
Enduring
Texas War
Aleutian
Yemen,
KhobarOp
South of
Philippines
Spanish Indian
World
USS
North
Dominican
Israel
Texas 1812
Campaign
D−Day
Lebanon
Of Korea
Grenada
Yangtze
Barbary
Franco−Amer
NorthChina Mexico
Atlantic
Riyadh,
EnduringSiberia
Border
Freedom,
Italy ● War
Wars
American
Trade
Towers,
Persian Cole
Gulf
Oklahoma
Desert
Campaigns
Boxer
Panama
Attack/USS
War War
Center
Saudi
City
Expedition
Rebellion
Liberty
Independence
Service
Wars
Civil
Naval
Nicaragua
Mexican
Haiti Expedition
Saudi
War
Freedom,
Cortina
Japan
Trieste Arabia
Shield/Storm
Afghanistan
War War
Arabia
War
Afghanistan
Theater
●
●
●●● ●
●
0e+00 1e+05 2e+05 3e+05 4e+05 5e+05
X: Numbers of American Soldiers Killed in Action
β̂1 = 1.23 −→ One additional soldier killed predicts 1.23 additional soldiers
wounded on average
Wounded (Scale in Levels)
World War II ●
Civil War, North ●
World War I ●
Vietnam War ●
Civil War, South ●
Korean War ●
Okinawa ●
Operation Iraqi Freedom, Iraq ●
Iwo Jima ●
Revolutionary War ●
War of 1812 ●
Aleutian Campaign ●
D−Day ●
Philippines War ●
Indian Wars ●
Spanish American War ●
Terrorism, World Trade Center ●
Yemen, USS Cole ●
Terrorism Khobar Towers, Saudi Arabia ●
Persian Gulf ●
Terrorism Oklahoma City ●
Persian Gulf, Op Desert Shield/Storm ●
Russia North Expedition ●
Moro Campaigns ●
China Boxer Rebellion ●
Panama ●
Dominican Republic ●
Israel Attack/USS Liberty ●
Lebanon ●
Texas War Of Independence ●
South Korea ●
Grenada ●
China Yangtze Service ●
Mexico ●
Nicaragua ●
Barbary Wars ●
Russia Siberia Expedition ●
China Civil War ●
Terrorism Riyadh, Saudi Arabia ●
North Atlantic Naval War ●
Franco−Amer Naval War ●
Operation Enduring Freedom, Afghanistan ●
Mexican War ●
Operation Enduring Freedom, Afghanistan Theater ●
Haiti ●
Texas Border Cortina War ●
Nicaragua ●
Italy Trieste ●
Japan ●
0e+00 1e+05 2e+05 3e+05 4e+05 5e+05 6e+05
Number of Wounded
Wounded (Logarithmic Scale)
Number of Wounded
10 100 1,000 10,000 100,000 1,000,000
World War II ●
Civil War, North ●
World War I ●
Vietnam War ●
Civil War, South ●
Korean War ●
Okinawa ●
Operation Iraqi Freedom, Iraq ●
Iwo Jima ●
Revolutionary War ●
War of 1812 ●
D−Day ●
Philippines War ●
Indian Wars ●
Spanish American War ●
Terrorism, World Trade Center ●
Yemen, USS Cole ●
Terrorism Khobar Towers, Saudi Arabia ●
Persian Gulf ●
Terrorism Oklahoma City ●
Persian Gulf, Op Desert Shield/Storm ●
Russia North Expedition ●
Moro Campaigns ●
China Boxer Rebellion ●
Panama ●
Israel Attack/USS Liberty ●
Lebanon ●
Texas War Of Independence ●
South Korea ●
Grenada ●
Mexico ●
Nicaragua ●
Barbary Wars ●
Russia Siberia Expedition ●
China Civil War ●
Terrorism Riyadh, Saudi Arabia ●
North Atlantic Naval War ●
Franco−Amer Naval War ●
Operation Enduring Freedom, Afghanistan ●
Mexican War ●
Operation Enduring Freedom, Afghanistan Theater ●
Haiti ●
Texas Border Cortina War ●
Nicaragua ●
Italy Trieste ●
Japan ●
2 4 6 8 10 12
Log(Number of Wounded)
Regression: Log-Level
World War II
●
Civil War, North
Y: log(Numbers of American Soldiers Wounded in Action)

World War I ●
Vietnam War ● Civil War, South
12
Korean●War ●
●
Okinawa
●
Operation Iraqi Freedom, Iraq
Iwo●Jima
10 ●
Revolutionary War
●
War of 1812
Aleutian ●
Campaign
D−Day
Philippines War
●●● Wars
8
Indian
Spanish American
● War
●
Terrorism, World Trade Center
●
Yemen, USS Cole
Terrorism Khobar Towers,
● Gulf Saudi Arabia
Persian
PersianTerrorism
Gulf, OpOklahoma
●
Desert City
Shield/Storm
Russia
Moro ● Expedition
North
Campaigns
6
●
● Rebellion
China Boxer
Panama
●
Dominican
Israel ● Republic
●
Attack/USS
Lebanon Liberty
Texas War Of●Independence
South Korea
●
Grenada
●
China Yangtze
● Service
Mexico
Nicaragua
Barbary
Russia ● Wars
Siberia Expedition
Dominican
Terrorism
NorthChina ●
● Republic
Riyadh,
Civil
Atlantic Saudi
War War
Naval Arabia
Franco−Amer
4
Operation Enduring ●
●
Freedom,
Mexican
●
Operation Enduring Freedom, WarAfghanistan
Haiti Afghanistan Theater ● ●
Texas Border ●
●Cortina War
Nicaragua
Italy Trieste
● ●
●
Japan
2
0e+00 1e+05 2e+05 3e+05 4e+05
X: Numbers of American Soldiers Killed in Action
β̂1 = 0.0000237 −→ One additional soldier killed predicts 0.0023 percent increase
in the number of soldiers wounded on average
Regression: Log-Log
World War II
●
Civil War, North
Y: Log(Numbers of American Soldiers Wounded in Action)

World War I ●
Vietnam War● War, South
Civil
12
● War
Korean ●
●
Okinawa
●
Operation Iraqi Freedom, Iraq
● Iwo Jima
10 ●
Revolutionary War
●
War of 1812
D−DayPhilippines War
● ●
8
Indian
● Wars
Spanish American●War
●
Terrorism, World Trade Center
● Yemen, USS Cole
Terrorism Khobar ●Towers, Saudi Arabia
Persian Gulf
● Terrorism Oklahoma
Persian Gulf, City Shield/Storm
Op Desert
Russia
Moro ●North
Campaigns Expedition
6
China Boxer Rebellion●

Panama ●
● ●
● ● Liberty
Dominican
Israel Attack/USS Republic Lebanon
● South
● KoreaTexas●
War Of Independence
Grenada ● ●
Mexico
Barbary Wars Nicaragua
● Russia Siberia Expedition
Terrorism Riyadh,
China Civil ●
Saudi
Franco−Amer War
Arabia Dominican
● North
Naval ● Republic
War Atlantic Naval War
4
● Operation
● ●● Afghanistan
Enduring Freedom,
●
Operation Enduring Freedom, Afghanistan ● Mexican War
● Haiti Theater ●
Texas Border Cortina War ● ●
Nicaragua
Italy●Trieste
●
●
Japan
2
2 4 6 8 10 12
X: Log(Numbers of American Soldiers Killed in Action)
β̂1 = 0.797 −→ A percent increase in deaths predicts 0.797 percent increase in

the wounded on average
References
Acemoglu, Daron, Simon Johnson, and James A. Robinson. “The colonial

origins of comparative development: An empirical investigation.” 2000.
Wooldridge, Jeffrey. 2000. Introductory Econometrics. New York:
South-Western.

Simple Linear Regression

Uploaded by

Copyright:

Available Formats

Simple Linear Regression

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Simple Linear Regression

Uploaded by

Copyright:

Available Formats

Week 5: Simple Linear Regression

October 10, 12, 2016

The next few weeks,

2 Properties of the OLS estimator

3 Example and Review

5 Hypothesis tests for regression

6 Confidence intervals for regression

8 Wrap Up of Univariate Regression

9 Fun with Non-Linearities

The (population) simple linear regression model can be stated as the

The estimated or sample regression function is:

rb(Xi ) = Ybi = βb0 + βb1 Xi

βb0 , βb1 are the estimated intercept and slope

You can think of the residuals as the prediction errors of our

Learn how to run and read regression

How do we derive the LS estimators for β0 and β1 ? We want to

Now we’re done! Here are the OLS estimators:

Where here we have the weights, Wi as:

2 Properties of the OLS estimator

3 Example and Review

5 Hypothesis tests for regression

6 Confidence intervals for regression

8 Wrap Up of Univariate Regression

9 Fun with Non-Linearities

1 Draw a random sample of size n = 30 with replacement using

Log Settler Mortality

Log Settler Mortality

What is the sampling distribution of the OLS slope?

We need fill in those ?s.

1 Linearity in Parameters: The population model is linear in its

1(./(%)$*/$*2* 1(./(%)$*/$*2* 1(./(%)$*/$*2* 1(./(%)$*/$*2*

7($")8*9(80:/$;* 7($")8*9(80:/$;* 7($")8*9(80:/$;*

</$#(./,6*/$* </$#(./,6*/$* </$#(./,6*/$*

>#.)*5)$"/%)$(:* >#.)*5)$"/%)$(:* >#.)*5)$"/%)$(:*

Note that it can be nonlinear in variables

β0 , β1 : Population parameters — fixed and unknown

Assumption (II. Random Sampling)

Data examples consistent with this assumption:

Why do we need this?

This assumption is needed just to calculate β̂, i.e. identifying β̂.

Assumption (IV. Zero Conditional Mean)

E [u|X ] = 0 implies a slightly weaker condition Cov(X , u) = 0

Example: Wage = β0 + β1 education + u. What is likely to be in u?

But let’s compare two situations:

Assumption 4 violated Assumption 4 not violated

Theorem (Unbiasedness of OLS)

E [β̂0 ] = β0 and E [β̂1 ] = β1

Now we know that, under Assumptions 1-4, we know that

That is we know that the sampling distribution is centered on the

In order to derive the sampling variance of the OLS estimator,

Assumption (V. Homoskedasticity)

This implies Var[u] = σu2

Violation: Var[u|X = x1 ] 6= Var[u|X = x2 ] called heteroskedasticity.

var[βb1 |X1 , . . . , Xn ] =??

where Var[u | X ] = σu2 (the error variance).

What drives the sampling variability of the OLS estimator?

ûi = yi − ŷi = yi − β̂0 − β̂1 xi

Intuitively, which line is likely to be closer to the observed sample values on X

Thus, an unbiased estimator for the error variance is:

1(./(%)$/$2* 1(./(%)$/$2* 1(./(%)$/$2* 1(./(%)$/$2*

7($")89(80:/$; 7($")89(80:/$; 7($")89(80:/$;

</$#(./,6/$ </$#(./,6/$ </$#(./,6/$

>#.)5)$"/%)$(: >#.)5)$"/%)$(: >#.)5)$"/%)$(:

1(./(%)$/$2* 1(./(%)$/$2* 1(./(%)$/$2* 1(./(%)$/$2*

7($")89(80:/$; 7($")89(80:/$; 7($")89(80:/$;

</$#(./,6/$ </$#(./,6/$ </$#(./,6/$

>#.)5)$"/%)$(: >#.)5)$"/%)$(: >#.)5)$"/%)$(: