6.434J/16.

391J
Statistics for Engineers and Scientists Apr 11
MIT, Spring 2006 Handout #13

Solution 5

Problem 1: Let a > 0 be a known constant, and let θ > 0 be a parameter.

Suppose X1 , X2 , . . . , Xn is a sample from a population with one of the following
densities.

(a) The beta, β(θ, 1), density: fX (x | θ) = θxθ−1 , for 0 < x < 1.
a
(b) The Weilbull density: fX (x | θ) = θaxa−1 e−θx , for x > 0.
θaθ
(c) The Pareto density: fX (x | θ) = x(θ+1)
, for x > a.

In each case, find a real-valued sufficient statistic for θ.

Solution Let X
(X1 , X2 , . . . , Xn ) be a collection of i.i.d. random variables

Xi ’s, and let x
(x1 , x2 , . . . , xn ) be a collection of observed data.

(a) For any x, the joint pdf is


θn (x x · · · x )θ−1 , if ∀i, 0 < xi < 1
1 2 n
fX (x | θ) =
0, otherwise;
= θn (x1 x2 · · · xn )θ−1 × I(0,1) (x1 )I(0,1) (x2 ) · · · I(0,1) (xn ) .
     

g(T (x) | θ)
h(x)

Factorization theorem implies that

T (x)
x1 x2 · · · xn

is a sufficient statistic for θ.

(b) For any x, the joint pdf is


θn an (x x · · · x )a−1 e−θ
 n
xa
1 2 n i=1 i , if ∀i, xi > 0;
fX (x | θ) =
0, otherwise;
 n
xa
= θ n e 
−θ i=1 i


g(T (x) | θ)

× an (x1 x2 · · · xn )a−1
×I(0,∞) (x1 )I(0,∞) (x2 ) · · · I(0,∞) (xn ) .
  

h(x)

1
 Factorization theorem implies that
n

T (x)
xai
i=1

is a sufficient statistic for θ.

(c) For any x, the joint pdf is


 θ n anθ
, if ∀i, xi > a;
(x1 x2 ···xn )θ+1
fX (x | θ) =
0, otherwise;
θn anθ
= × I(a,∞) (x1 )I(a,∞) (x2 ) · · · I(a,∞) (xn ) .
(x1 x2 · · · xn )θ+1   
  

h(x)

g(T (x) | θ)

Factorization theorem implies that

T (x)
x1 x2 · · · xn

is a sufficient statistic for θ.

Problem 2:

a) Let X1 , X2 , . . . , Xn be independent random variables, each uniformly dis-

tributed on the interval [−θ, θ], for some θ > 0. Find a sufficient statistic
for θ.

b) Let X1 , X2 , . . . , Xn be a random sample of size n from a normal N (θ, θ)

distribution, for some θ > 0. Find a sufficient statistic for θ.

Solution

a) For any x
(x1 , x2 , . . . , xn ), the joint pdf is given by


n
 1 , if ∀i, −θ ≤ xi ≤ θ;
2θ
fX (x | θ) =
0, otherwise;

n
 1 , if −θ ≤ min(x1 , . . . , xn ) and max(x1 , . . . , xn ) ≤ θ;
2θ
=
0, otherwise;
1
n
= I[−θ,∞) (min(x1 , . . . , xn ))I(−∞,θ] (max(x1 , . . . , xn )) × 
1 .
 2θ  
h(x)

g(T(x) | θ)

2
 Factorization theorem implies that

T(x)
min(x1 , . . . , xn ), max(x1 , . . . , xn )

is jointly sufficient for θ.

b) For any x
(x1 , x2 , . . . , xn ), the joint pdf is given by

1
n 1 n  2
fX (x) = √ e− 2θ i=1 (xi −θ)
 
2πθ
1
n 1 n 2 n 2
= √ e− 2θ ( i=1 xi −2θ i=1 xi +nθ )
 
2πθ
1
n 1 n 2 n nθ
= √ e− 2θ i=1 xi + i=1 xi − 2
 
2πθ
1
n n
1
n 1 n 2 nθ
= √ e i=1 xi × √ e− 2θ i=1 xi − 2 .
2π θ
     

h(x)
g(T (x) | θ)

Factorization theorem implies that

n

T (x)
x2i
i=1

is a sufficient statistic for θ.

Problem 3: Let X be the number of trials up to (and including) the first

success in a sequence of Bernoulli trials with probability of success θ, for 0 <
θ < 1. Then, X has a geometric distribution with the parameter θ:

Pθ {X = k} = (1 − θ)k−1 θ, k = 1, 2, 3, . . . .

Show that the family of geometric distributions is a one-parameter exponential

family with T (x) = x.
[Hint: xα = eα ln x , for x > 0.]
Solution Recall that the pmf of a one-parameter (θ) exponential family is of
the form

p(x | θ) = h(x) eη(θ)T (x)−B(θ) ,

where x ∈ X . Rewriting the pmf of a Geometric random variable yields

Pθ {X = x} = e(x−1) ln(1−θ)+ln θ
= ex ln(1−θ)−(ln(1−θ)−ln θ) ,

3
where x ∈ {1, 2, 3, . . . }. Thus, the geometric distribution is a one-parameter
exponential family with

h(x)
1 η(θ)
ln(1 − θ)

T (x)
x B(θ)
ln(1 − θ) − ln θ
X
{1, 2, 3, . . . }.

Problem 4: Let X1 , X2 , . . . , Xn be a random sample of size n from the trun-

cated Bernoulli probability mass function (pmf),

p, if x = 1;
P {X = x | p} =
(1 − p), if x = 0.

(a) Show that the joint pmf of X1 , X2 , . . . , Xn is a member of the exponential

family of distribution.

(b) Find a minimal sufficient statistic for p.

Solution

(a) Let x
(X1 , X2 , . . . Xn ) denote the collection of i.i.d. Bernoulli random

variables. The joint pmf is given by
    
P {X = x | p} = px1 (1 − p)1−x1 px2 (1 − p)1−x2 · · · pxn (1 − p)1−xn
 n
xi
 n
=p (1 − p)n− xi
 
i=1 i=1

n n
(ln p) xi
e[ln(1−p)][n− xi ]
=e

i=1 i=1

n
[ln p−ln(1−p)] xi +n ln(1−p)
=e i=1 ,

for x ∈ {0, 1}n . Therefore, the joint pmf is a member of the exponential
family, with the mappings:

θ=p h(x) = 1
n

η(p) = ln p − ln(1 − p) T (x) = xi
i=1

B(p) = −n ln(1 − p) X = {0, 1}n .

(b) Let x, y ∈ {0, 1}n be given. Consider the likelihood ratio,

P {X = x | p}  n 
n
= e[ln p−ln(1−p)][ i=1 xi − i=1 yi ]
.
P {X = y | p}

4
 Define a function k(x, y)
h(x)/h(y) = 1, which is bounded and non-zero
for any x ∈ X and y ∈ X .
n n
Note that x and y such that i=1 xi = i=1 yi are equivalent because
function k(x, y) satisfies the requirement of likelihood ratio partition.
n
Therefore, T (x)
i=1 xi is a sufficient statistic.

Problem 5: Let X1 , X2 , . . . , Xm and Y1 , Y2 , . . . , Yn be two independent sam-

ples from N (µ, σ 2 ) and N (µ, τ 2 ) populations, respectively. Here, −∞ < µ < ∞,
σ 2 > 0, and τ 2 > 0. Find a minimal sufficient statistic for θ
(µ, σ 2 , τ 2 ).

Solution Let X
(X1 , X2 , . . . , Xm ) and Y
(Y1 , Y2 , . . . , Yn ) denote the col-

lections of random samples. The joint pdf (of Xj ’s and Yi ’s), evaluated at
x
(x1 , x2 , . . . , xm ) and y
(y1 , y2 , . . . , yn ), is given by
1
m m (xj −µ)2 
j=1 1
n − ni=1 (y2i −µ)2
fX,Y (x, y | θ) = √ · e− 2σ 2 · √ ·e 2τ
2πσ 2
1

m
x2j − 2τ12

2πτ 2
n
yi2 + σµ2
 m
xj + τµ2

n
yi −B(µ,σ 2 ,τ 2 )
= e− 2σ2 j=1 i=1 j=1 i=1 ,
2 2
mµ nµ
where B(µ, σ 2 , τ 2 )
m 2 n 2
2 ln 2πσ + 2 ln 2πτ + 2σ 2 + 2τ 2 .
Notice that the joint pdf belongs to the exponential family, so that the
minimal statistic for θ is given by

m n
 m
 n


T(X, Y)
Xj2 , Yi2 , Xj , Yi .

j=1 i=1 j=1 i=1

Note: One should not be surprised that the joint pdf belongs to the exponen-
tial family of distribution. Recall that Gaussian distribution is a member of the
exponential family of distribution and that random variables, Xi ’s and Yj ’s, are
mutually independent. Thus, their joint pdf belongs to the exponential family
as well.
Note: To derive the minimal sufficient statistic, one may alternatively consider
likelihood ratio partition.
The set D0 is defined to be
 

D0
(x, y) ∈ Rm+n  for all µ, for all σ 2 > 0, for all τ 2 > 0
  
fX,Y x, y | µ, σ 2 , τ 2 = 0

=∅ (empty set).

5
Let (x, y) ∈
/ D0 and (v, w) ∈
/ D0 be given. Their likelihood ratio is given by

1  2  2
1  2  2

m m n n
fX,Y (x, y | θ)
= exp − 2 xj − vj − 2 yi − wi
fX,Y (v, w | θ) 2σ j=1 j=1
2τ i=1 i=1

µ 
m m
µ 
n n

+ 2 xj − vj + 2 yi − wi .
σ j=1 j=1
τ i=1 i=1

By definition, (x, y) ∈ / D0 and (v, w) ∈/ D0 are equivalent iff there exists a

function, 0 < k(·, ·, ·, ·) < ∞, which is independent of θ, such that

fX,Y (x, y | θ)
= k(x, y, v, w).
fX,Y (v, w | θ)

The likelihood ratio implies that (x, y) ∈

/ D0 and (v, w) ∈
/ D0 are equivalent if
and only if
m
 m

x2j = vj2 , (1)
j=1 j=1
n n
yi2 = wi2 , (2)
i=1 i=1
m m
xj = vj , and (3)
j=1 j=1
n n
yi = wi , (4)
i=1 i=1

where function k(x, y, v, w)
1.

That is, (x, y) and (v, w) are in the same equivalent class iff conditions
(1)-(4) are satisfied. Then a representation of the equivalent class is given by

m n
 m
 n


T(X, Y)
Xj2 , Yi2 , Xj , Yi .

j=1 i=1 j=1 i=1

Thus, we have a minimal sufficient statistic, T(X, Y).

Problem 6: The two hypotheses about the probability density fX (x) of an

observed random variable X are
1 −|x|
H1 : fX (x) = e , for any x
2
1 1 2
H0 : fX (x) = √ e− 2 x , for any x.
2π

6
 (a) Find the likelihood ratio Λ(x).

(b) The test is of the form

H1
Λ(x) ≷ η .
H0

Compute the decision regions for various values of the threshold η.

Solution

(a) Let x ∈ R denote an observation. The likelihood ratio is given by

fX | H (x | H1 )
Λ (x)
.
fX | H (x | H0 )

Substituting the densities of random variable X (under hypothesis H1 and

under hypothesis H0 ) yields the likelihood ratio

1 −|x|
2 e
Λ (x) = 1 2

√1 e− 2 x
2π

π 1 x2 −|x|
= e2 .
2

(b) The decision region for hypothesis H1 , R1 , is the set of points x’s that
give rise to the output decision H1 :

R1
{x | the test decides H1 on input x}

= {x | Λ (x) > η}.

Similarly, the decision region for hypothesis H0 , R0 , is given by

R0
{x | the test decides H0 on input x}

= {x | Λ (x) ≤ η}
= R\R1 ,

where the symbol “\” denotes the set difference.

Substituting the expression of the likelihood ratio from part (a) yields the
following definition of decision region R1 :
  
 π 1 x2 −|x|
R1 = x  e2 >η .
2
When 0 ≥ η, we will have R1 = R since ey > 0 ≥ η for any y. Thus, we
will consider the case when η > 0.

7
Taking natural log both sides of the inequality and writing x2 as |x|2 yields
  
1 η √2

 2
R1 = x  |x| − |x| − ln √ >0 .
2 π
√

When 1 + 2 ln η√π2 < 0, the decision region is empty (since the term,
b2 − 4ac, in the square root of the quadratic formula is negative).
√
π −1
When 1 + 2 ln η√π2 ≥ 0, or equivalently, η ≥ 2 e
2 , we will have a

non-empty decision region,


 
 √2


R1 = x  |x| > 1 + 1 + 2 ln η or
π

√2

|x| < 1 − 1 + 2 ln η
π
 √
  2


= x  |x| > 1 + 1 + 2 ln η
π
(absolute value cannot be negative)

 
 √2

= x  x > 1 + 1 + 2 ln η or
π

√2

x < −1 − 1 + 2 ln η .
π

Therefore, the decision region R1 is given by



R, for η ≤ 0;



 π
∅, for 0 < η < e−1/2 ;
R1 =  √ √
2

 (1 + 1 + 2 ln(η 2/ π), ∞)

 

(−∞, 1 − 1 + 2 ln(η √2/√π) ),

for
π
e−1/2 ≤ η,
2

while the decision region R0 is given by

R0 = R\R1


∅, for η ≤ 0;



 
R, for 0 < η < π2 e−1/2 ;
=   √ √

 1 − 1 + 2 ln(η 2/ π),

  

 1 + 1 + 2 ln(η √2/√π) , for  π e−1/2 ≤ η.

2