Good Estimation Detection Notes
Good Estimation Detection Notes
Good Estimation Detection Notes
March 2, 2006
Contents
1 Problem Statement 2
5 Further problems 8
5.1 Problem 1: relations for Linear Gaussian Models . . . . . . . . . . . . . . . . . . . . . . . . . . 8
5.1.1 No a priori information: MMSE and LS . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5.1.2 Uncorrelated noise: BLU and LS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5.1.3 Zero-noise: MMSE and LS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5.2 Problem 2: orthogonality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5.3 Practical example: multi-antenna communication . . . . . . . . . . . . . . . . . . . . . . . . . 9
5.3.1 Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5.3.2 Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1
1 Problem Statement
We are interested in estimating a continuous scalar1 parameter a ∈ A from a vector observation r. The
observation and the parameter are related through a probabilistic mapping pR|A (r |a ). a may or may not
be a sample of a random variable. This leads to Bayesian and non-Bayesian estimators, respectively.
Special cases
• Noisy observation model:
r = f (a, n)
where n (the noise component) is independent of a, and has a pdf pN (n).
The latter two will be covered in this section. We remind that the MAP estimate is given by
Taking the derivative (assuming this is possible) w.r.t. â (r) and setting the result to zero yields
Z
â (r) = a × pA|R (a |r ) da.
A
1 We will only consider estimation of scalars. Extension to vectors is straightforward.
2
2.1.2 Important special case: Gaussian model
When A and R are jointly Gaussian, we can write
[A, R] ∼ N (m, Σ)
where · ¸
mA
m=
mR
and · ¸
ΣA ΣTAR
Σ= .
ΣAR ΣR
In that case it can be shown (after some straightforward manipulation), that
¡ ¢
A|R ∼ N mA + ΣTAR Σ−1 T −1
R (r − mR ) , ΣA − ΣAR ΣR ΣAR
so that
âM M SE (r) = mA + ΣTAR Σ−1
R (r − mR ) (1)
Note that
• A posteriori (after observing r), the uncertainty w.r.t. a is reduced from ΣA to ΣA −ΣTAR Σ−1
R ΣAR < ΣA .
ΣAR = E {(R − mR ) (A − mA )}
= E {(hA + N − hmA ) (A − mA )}
= E {(h (A − mA ) + N) (A − mA )}
n o
2
= hE (A − mA ) + E {N (A − mA )}
= hΣA
and
n o
T
ΣR = E (R − mR ) (R − mR )
n o
T
= E (h (A − mA ) + N) (h (A − mA ) + N)
n o © ª
2
= hE (A − mA ) hT + E NNT
= hΣA hT + ΣN .
3
This leads to
â (r) = bT r + c
so that â (r) is obtained through an affine transformation of the observation. Clearly, when A and R are
jointly Gaussian, the MMSE estimator is a linear estimator with
bT = ΣTAR Σ−1
R
and
c = mA + ΣTAR Σ−1
R (mR ) .
bT mR + c = mA
c = mA − bT mR
c2 = m2A + bT mR mTR b − 2mA bT mR
n o
T
ΣR = E (R − mR ) (R − mR )
© ª
= E RRT − mR mTR
and
ΣAR = E {(R − mR ) (A − mA )}
= E {RA} − mR mA .
4
Let us now look at the variance of the estimation error (the estimation error is zero-mean):
n o
2
V = E (â (R) − A)
n¡ ¢2 o
= E bT R + c − A
© ª © ª
= bT E RRT b + E A2 + c2 + 2cbT mR − 2cmA − 2bT E {RA}
¡ ¢ ¡ ¢
= bT ΣR + mR mTR b + ΣA + m2A + c2 + 2c bT mR − mA − 2bT (ΣAR + mR mA )
¡ ¢
= bT ΣR + mR mTR b + ΣA + m2A − c2 − 2bT (ΣAR + mR mA )
= bT ΣR b + ΣA − 2bT ΣAR
where we have used the fact that c2 = m2A + bT mR mTR b − 2mA bT mR . Taking derivative w.r.t. b and
equating the result to zero, yields2
2ΣR b − 2ΣAR = 0
and thus (since Σ−1
R is symmetric)
b = Σ−1
R ΣAR (3)
and
c = mA − bT mR
= mA − ΣTAR Σ−1
R mR (4)
Substitution of (3) and (4) into (2) gives us the final result
QED.
5
n ³ ¡ ¢−1 ´o
E {Râ (R)} = E R ΣA hT hΣA hT + ΣN R
© ª¡ ¢−1
= E RRT hΣA hT + ΣN ΣA h
¡ T
¢¡ T
¢−1
= hΣA h + ΣN hΣA h + ΣN ΣA h
= ΣA h
The latter two will be covered in this section. All expectations are taken for a given value of a. We remind
that the ML estimate is given by
âM L (r) = arg max pR|A (r |a ) .
a∈A
â (r) = bT r + c.
E {R} = ha
which is assumed to be known to the estimator. Our goal is to find a b that leads to an unbiased estimator
with minimal variance for all a.
Unbiased
An unbiased estimator must satisfy E {â (r)} = a, for all a ∈ A, so that
bT h = 1.
6
Variance of Estimation Error
The variance of the estimation error is given by
n o
2
Va = E (â (R) − a)
n¡ ¢2 o
= E bT r − a
n¡ ¢2 o
= E bT (r − ha)
n o
2
= bT E (r − ha) b
= bT ΣR b.
This leads us to the following optimization problem: find b that minimizes Va , subject to bT h = 1. Using a
Lagrange multiplier technique, we find we need to minimize
¡ ¢
bT ΣR b − λ bT h − 1 .
This leads to
b = Σ−1
R hλ.
λhT Σ−1
R h=1
so that ¡ T −1 ¢−1
b = Σ−1
R h h ΣR h
and
¡ ¢−1 ¡ T −1 ¢−1
Va = hT Σ−1
R h hT Σ−1 −1
R ΣR ΣR h h ΣR h
¡ ¢−1
= hT Σ−1
R h .
Hence ¡ ¢−1 T −1
âBLU (r) = hT Σ−1
N h h ΣN r
7
3.2 The LS estimator
If we refer back to our problem statement, the LS estimator tries to find a that minimizes the distance the
between the observation and the ‘reconstructed’ noiseless observation:
n o
2
d = E kr − h (a, 0)k .
³ ¡ ¢−1 T ´T
T
(r − hâLS (r)) a = r − h hT h h r a
T
= (r − r) a
= 0
¡ ¢−1
âLM M SE (r) = mA + ΣA HT HΣA HT + ΣN (r − HmA )
¡ T −1 ¢
−1 −1 T −1
= mA + H ΣN H + ΣA H ΣN (r − HmA )
¡ ¢−1 T
âLS (r) = HT H H r
¡ T −1 ¢−1 T −1
âBLU (r) = H ΣN H H ΣN r
5 Further problems
5.1 Problem 1: relations for Linear Gaussian Models
How are MMSE, LS and BLU related? Consider the scalar case. The estimators are given by
¡ ¢−1
âM M SE (r) = mA + ΣA hT hΣA hT + ΣN (r − hmA )
¡ ¢−1 T
âLS (r) = hT h h r
¡ T −1 ¢−1 T −1
âBLU (r) = h ΣN h h ΣN r
8
5.1.1 No a priori information: MMSE and LS
When mA = 0 and Σ−1
A = 0 we get
¡ ¢−1
âM M SE (r) = hT hhT r
¡ T ¢−1 T
= hh h r
= âLS (r)
so that ¡ ¢−1
b = hΣA hT + ΣN ΣA h.
Hence ¡ ¢−1
q̂LM M SE (s) = ΣA hT hΣA hT + ΣN s.
The estimate of A is then given by
9
5.3.1 Problem
We are interested in the following problem: we transmit a vector of nT iid data symbols a ∈ A = ΩnT over
an AWGN channel. Here nT is the number of transmit antennas. The receiver is equipped with nR receive
antennas. We can write the observation as
r = Ha + n
¡ ¢
where H is a (known) nR × nT channel matrix and N ∼ N 0, σ 2 InR . In any practical communication
scheme, Ω is a finite set of size M (e.g., BPSK signaling with Ω = {−1, +1}). The symbols are iid, uniformly
distributed over Ω.
1. Determine the ML and MAP estimates of a. Determine the complexity of the receiver (number of
operations to estimate a) as a function of nT .
2. How can we use LMMSE and LS be used to estimate a? We assume ΣA = InT and mA = 0. Determine
the complexity of the resulting estimators as a function of nT .
5.3.2 Solution
Solution - part 1
The MAP and ML estimators are considered to be optimal in the case of estimating discrete parameters, in
a sense of minimizing the error probability.
= arg max
n
log pR|A (r |a )
a∈Ω T
1 2
= arg max − kr − Hak
n
a∈Ω σ2T
2
= arg min
n
kr − Hak
a∈Ω T
pA (a)
= arg max pR|A (r |a )
n
a∈Ω T pR (r)
= arg max
n
pR|A (r |a )
a∈Ω T
= âM L (r)
2
Complexity: for each a ∈ ΩnT , we need to compute kr − Hak . Hence, the complexity is exponential in the
number of transmit antennas.
Special case:
Let us assume Ω = {−1, +1} and nT = 1, nR > 1. Then
¡ ¢
âM L (r) = arg max ahT r
a∈Ω
In this case we combine the observations from multiple receive antennas. Each observation is weighted
with the channel gain. This means that when the channel is unreliable on a given antenna (so that hk is
small), this has only a small contribution to our decision statistic.
10
Solution - part 2
Let us temporarily forget that a lives in a discrete space. We can then introduce the LMMSE and LS esti-
mates as follows:
¡ ¢−1
âLM M SE (r) = HT HHT + σ 2 InR r
¡ ¢−1 T
âLS (r) = HT H H r.
Note that âLM M SE (r) and âLS (r) may not belong to ΩnT . Hence, we need to quantize these estimates
(mapping the estimate to the closest point in ΩnT ); this can be done on a symbol-per-symbol basis. Gen-
erally the LS estimator will have poor performance when HHT is close to singular. Note that the LMMSE
estimator requires knowledge of σ 2 , while the MAP and ML estimators do not.
Complexity: now the complexity is of the order nR × nT (since any matrix not depending on r can be
pre-computed by the receiver), which is linear in the number of transmit antennas.
Special case: -=-=-Comment: there was a mistake in the lecture. These are the correct results-=-=-
Let us assume that we use more transmit that receive antennas: nT > nR . In that case the the nR × nR
matrix HHT is necessarily singular, so that LS estimation will not work. The LMMSE estimator requires
the inversion of HHT + σ 2 InR . When σ 2 6= 0, this matrix is always invertible. This is a strange situation,
where noise is actually helpful in the estimation process.
11