Summary Statistics (FEB21007X)

Week 1
Random sampling assumption
Independent and identically distributed (i.i.d) random variables 𝑋! , … , 𝑋" with pdf 𝑓(𝑥; 𝜃),
where the parameter 𝜃 is unknown. 𝜃 lies in some parameter space Ω
Type of means
Population mean: the mean that is a characteristic of the type of the distribution
!
Sample mean: the observed mean, calculated as 𝑋+ = " ∑"#$! 𝑋#
Variables/value
The unknown/unobserved are random variables in uppercase, 𝑋! , … , 𝑋" ~(𝑖. 𝑖. 𝑑) 𝑓(𝑥; 𝜃)
The data consists of observed values in lower case from the random process, 𝑥! , … , 𝑥"
Estimates/estimate
An estimator 𝑇 = 𝑡(𝑋! , … , 𝑋" ) is a function of the random variables 𝑋! , … , 𝑋" , so also a rv
An estimate 𝑇 = 𝑡(𝑥! , … , 𝑥" ) is the observed value of the estimator 𝑇 (the function value) based
on the observed values 𝑥! , … , 𝑥"
Method of moments estimator (MME)
! %
Population moments: 𝐸6𝑋% 7 = 𝜇%& (𝜃! , … , 𝜃' ) and sample moments: " ∑"#$! 𝑋# = 𝑀%&
Obtain an estimator of the unknown parameter 𝜽 = (𝜃! , … , 𝜃' ) by equating population
moments to sample moments
Maximum likelihood estimator (MLE)
Use the value of the unknown parameter 𝜽 = (𝜃! , … , 𝜃' ) that is most likely to have generated
the observed data as estimate. The likelihood function is defined as 𝐿(𝜽) = 𝑓(𝑋! , … , 𝑋" ; 𝜽)
For a random sample 𝑋! , … , 𝑋" we have 𝐿(𝜽) = ∏"#$! 𝑓(𝑋# ; 𝜽)
The MLE 𝜽= = 6𝜃>! , … , 𝜃>' 7 is the value of 𝜽 that maximizes 𝐿(𝜽)
Maximum of 𝐿(𝜽)
(
Compute MLE as the solution of () 𝐿(𝜽) = 0, but it is not always possible to compute
(
∏"#$! 𝑓(𝑋# ; 𝜽). Taking the logarithm of 𝐿(𝜽) gives the log-likelihood ln6𝐿(𝜽)7, then compute
()
( (!
()
ln6𝐿(𝜽)7 = 0 and check if ()! ln6𝐿(𝜽)7 < 0. If 𝐿(𝜽) is not differentiable, mathematical
reasoning is required
Invariance properties
If 𝜃> is the MLE of 𝜃 and if 𝑢(𝜃) is a function of 𝜃, then 𝑢6𝜃>7 is an MLE of 𝑢(𝜃)
= = 6𝜃>! , … , 𝜃>' 7 denotes the MLE of 𝜽 = (𝜃! , … , 𝜃' ), then the MLE of
If 𝜽
𝜏(𝜽) = 6𝜏! (𝜽), … , 𝜏* (𝜽)7 is 𝜏6𝜽 =7 = E𝜏! 6𝜽
=7, … , 𝜏* 6𝜽
= 7F for 1 ≤ 𝑟 ≤ 𝑘
Unbiased estimator
An estimator 𝑇 is said to be an unbiased estimator of 𝜏(𝜃) if 𝐸(𝑇) = 𝜏(𝜃) for all 𝜃 ∈ Ω.
Otherwise, we say that 𝑇 is a biased estimator of 𝜏(𝜃)
The bias is defined as 𝐵𝑖𝑎𝑠(𝑇) = 𝐸(𝑇) − 𝜏(𝜃). With an unbiased estimator, on average the true
value is estimated (an accurate estimator)
Mean squared error (MSE)
+
If 𝑇 is an estimator of 𝜏(𝜃), then the MSE of 𝑇 is given by 𝑀𝑆𝐸(𝑇) = 𝐸6𝑇 − 𝜏(𝜃)7
+
Moreover, 𝑀𝑆𝐸(𝑇) = 𝑉(𝑇) + 6𝐵𝑖𝑎𝑠(𝑇)7 , so if 𝑇 is unbiased, the 𝑀𝑆𝐸(𝑇) = 𝑉(𝑇)
Uniformly minimum variance unbiased estimator (UMVUE)
An estimator 𝑇 ∗ of 𝜏(𝜃) is called a UMVUE of 𝜏(𝜃) is 𝑇 ∗ is unbiased for 𝜏(𝜃) and for any other
unbiased estimator 𝑇 of 𝜏(𝜃), 𝑉(𝑇 ∗ ) ≤ 𝑉(𝑇) for all 𝜃 ∈ Ω
Cramer-Rao lower bound (CRLB)
!
-." ())1
If 𝑇 is an unbiased estimator of 𝜏(𝜃), then the CRLB is 𝑉(𝑇) ≥ # ! or
"23 4567(8;)):;
#$

, !
-." ())1
𝑉(𝑇) ≥ #!
. If an unbiased estimator equals the CRLB, it is a UMVUE
<"2= ! 4567(8;)):>
#$
Relative efficiency
The relative efficiency of an unbiased estimator 𝑇 of 𝜏(𝜃) to another unbiased estimator 𝑇 ∗ of
?(@ ∗ )
𝜏(𝜃) is given by 𝑟𝑒(𝑇, 𝑇 ∗ ) = ?(@)
Efficiency
An unbiased estimator 𝑇 ∗ of 𝜏(𝜃) is said to be efficient if 𝑟𝑒(𝑇, 𝑇 ∗ ) ≤ 1 for all unbiased
estimators 𝑇 of 𝜏(𝜃), and all 𝜃 ∈ Ω. The efficiency of an unbiased estimator 𝑇 of 𝜏(𝜃) is given by
𝑒(𝑇) = 𝑟𝑒(𝑇, 𝑇 ∗ ) if 𝑇 ∗ is an efficient estimator of 𝜏(𝜃).
An unbiased estimator that reaches the CRLB is efficient and an efficient estimator is UMVUE
Estimator with several unknown parameters
If 𝑋! , … , 𝑋" have pdf 𝑓(𝑥; 𝜃! , … , 𝜃' ), solve the system of equations with 𝑘 equations.
! !
For MME, the system is: 𝐸(𝑋! ) = " ∑"#$! 𝑋#! , … , 𝐸(𝑋 ' ) = " ∑"#$! 𝑋#'
For MML, the system is all the 𝑘 partial derivatives of the log-likelihood equal to 0

Week 2
Simple consistency
Let {𝑇" } be a sequence of estimators of 𝜏(𝜃). These estimators are said to be consistent
estimators of 𝜏(𝜃) if for every 𝜀 > 0, lim 𝑃(|𝑇" − 𝜏(𝜃)| < 𝜀) = 1 for every 𝜃 ∈ Ω
"→B
C
This is equivalent to lim 𝑃(|𝑇" − 𝜏(𝜃)| > 𝜀) = 0. The notation is 𝑇" → 𝜏(𝜃)
"→B
A consistent estimator converges in probability to the true value
MSE consistency
If {𝑇" } be a sequence of estimators of 𝜏(𝜃), then they are called MSE consistent if
+
lim 𝐸6𝑇" − 𝜏(𝜃)7 = 0 for every 𝜃 ∈ Ω. MSE consistency ⟹ (simple) consistency
"→B
Asymptotically unbiased estimator
A sequence of estimators {𝑇" } is said to be asymptotically unbiased for 𝜏(𝜃) if lim 𝐸(𝑇" ) = 𝜏(𝜃)
"→B
for all 𝜃 ∈ Ω. Estimator is MSE consistent ⟺ it is asymptotically unbiased and lim 𝑉( 𝑇" ) = 0
"→B
Law of Large Numbers (LLN)
If 𝑋! , … , 𝑋" is a random sample from a distribution with finite mean and variance, then
! C
𝑋+" = " ∑"#$! 𝑋# → 𝐸(𝑋)
Continuous mapping theorem
C C
If 𝑌" → 𝑐, then for any function 𝑔(𝑦) that is continuous at 𝑐, 𝑔(𝑌" ) → 𝑔(𝑐)
This is also valid for 𝑘-dimensional vectors
Convergence in probability results theorem
C C
If 𝑋" and 𝑌" are two sequences of random variables such that 𝑋" → 𝑐 and 𝑌" → 𝑑, then:
C
- 𝑎𝑋" + 𝑏𝑌" → 𝑎𝑐 + 𝑏𝑑
C
- 𝑋" 𝑌" → 𝑐𝑑
C
- 𝑋" /𝑐 → 1 for 𝑐 ≠ 0
C
- 1/𝑋" → 1/𝑐 if 𝑐 ≠ 0
C
- g𝑋" → √𝑐 if 𝑐 > 0