Summary Probability Theory (FEB21005X)

Week 1
Discrete sample space
Finite or countably infinite number of outcomes
Continuous sample space
Uncountable number of outcomes
Axioms of probability
1. 𝑃(𝐴) ≥ 0 for any event A
2. 𝑃(𝑆) = 1
3. For any countable collection of disjoint events 𝑃(⋃" "
!#$ 𝐴! ) = ∑!#$ 𝑃(𝐴! )
Conditional probability
The conditional probability of an event A, given the event B, is defined by
%('∩))
𝑃(𝐴|𝐵) = %()) if 𝑃(𝐵) > 0
Bayes’ rule
Let 𝐴$ , … , 𝐴+ be disjoint and exhaustive events and assume 𝑃(𝐵) > 0, then
%()|'! )%('! )
𝑃2𝐴, 3𝐵4 = ∑#
"$% %()|'" )%('" )
Independence of A and B
Two events A and B are called independent events if 𝑃(𝐴 ∩ 𝐵) = 𝑃(𝐴)𝑃(𝐵)
The definitions 𝑃(𝐴 | 𝐵) = 𝑃(𝐴) and 𝑃(𝐵 | 𝐴) = 𝑃(𝐵) are equivalent
Mutually independent
Events 𝐴$ , … , 𝐴+ are mutually independent if for every 𝑘 = 2, 3, … , 𝑛 and for every subset
{𝑖$ , … , 𝑖/ } of {1, 2, … , 𝑛} 𝑃 =⋂/,#$ 𝐴!! ? = ∏/,#$ 𝑃 =𝐴!! ?
Random variable (rv)
A random variable is a function: 𝑆 → ℝ, we use capital letters to denote a rv
Discrete random variable
The set of possible values for the random variable is finite or countably infinite
The probability distribution of a discrete random variable is completely described by the
probability density function (pdf), defined by
𝑝(𝑥) = 𝑃(𝑋 = 𝑥) = 𝑃({𝑠 ∈ 𝑆: 𝑋(𝑠) = 𝑥}), for every number 𝑥
A function 𝑝(𝑥) is a discrete pdf if and only if 𝑝(𝑥! ) ≥ 0, ∀𝑥! en ∑122 3" 𝑝(𝑥! ) = 1
The cumulative distribution function (cdf) of a discrete rv is the function
𝐹(𝑥) = 𝑃(𝑥 ≤ 𝑥) = ∑4:463 𝑝(𝑦)
A discrete function is continuous from the left, so the probability that 𝑥 is between 𝑎 and 𝑏
is equal to 𝑃(𝑎 < 𝑥 ≤ 𝑏)
Continuous random variable
The set of all possible values for the random variable is uncountably infinite
The cdf of a continuous random variable X is a continuous function
A probability density function (pdf) of a continuous random variable X is a function 𝑓 such
3
that 𝐹(𝑥) = 𝑃(𝑋 ≤ 𝑥) = ∫7" 𝑓(𝑠)𝑑𝑠
If X is a continuous random variable with probability density function 𝑓 and cumulative
distribution function 𝐹, then at every x where 𝐹 8 (𝑥) exists, 𝐹′(𝑥) = 𝑓(𝑥)
Calculation probability continuous rv
9
For any 𝑎 ≤ 𝑏, 𝑃(𝑎 ≤ 𝑋 ≤ 𝑏) = ∫1 𝑓(𝑠)𝑑𝑠 = 𝐹(𝑏) − 𝐹(𝑎)
Writing down pdf
All distributions, except for the normal distribution, have a 0 part, so the pdf will be
…, 𝑓𝑜𝑟 …
𝑓(𝑥) = U
0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒


1

, Expectation
If X is a random variable with pdf 𝑓(𝑥) and 𝑢(𝑥) is a real-valued function whose domain
includes the possible values of X, then
- 𝐸2𝑢(𝑋)4 = ∑3 𝑢(𝑥)𝑓(𝑥) if X is discrete
"
- 𝐸2𝑢(𝑋)4 = ∫7" 𝑢(𝑥)𝑓(𝑥)𝑑𝑥 if X is continuous
To calculate the mean/expectation of X, we take 𝑢(𝑥) = 𝑥
Properties of expectation
- 𝐸(𝑐) = 𝑐
- 𝐸(𝑎𝑋 + 𝑏) = 𝑎𝐸(𝑋) + 𝑏
Variance
:
The variance of a random variable X is given by 𝑉(𝑋) = 𝐸 a2𝑋 − 𝐸(𝑋)4 b
Properties of variance
- 𝑉(𝑋) ≥ 0
- 𝑉(𝑎𝑋) = 𝑎: 𝑉(𝑋)
- 𝑉(𝑋 + 𝑏) = 𝑉(𝑋)
- 𝑉(𝑎𝑋 + 𝐵) = 𝑎: 𝑉(𝑋)
:
- 𝑉(𝑋) = 𝐸(𝑋 : ) − 2𝐸(𝑋)4
(Central) moment
The 𝑘;< moment 𝜇8 / of a random variable X is 𝐸(𝑋 / )
The 𝑘;< central moment 𝜇/ of a random variable X is 𝐸((𝑋 − 𝜇)/ )
=& ='
skewness = >& (measure of asymmetry) and kurtosis = >' (measure of fatness of tails)
Markov’s inequality
For a random variable that only takes positive values, it holds that, for every 𝑐 > 0,
?(3)
𝑃(𝑋 ≥ 𝑐) ≤ @
Chebyshev inequality
For every random variable X with expected value 𝜇 and variance 𝜎 : > 0 and 𝑘 > 0 it holds
>( >(
that 𝑃(|𝑋 − 𝜇| ≥ 𝑐) ≤ @ ( ⇔ 𝑃(|𝑋 − 𝜇| ≤ 𝑐) ≥ 1 − @ ( , or with 𝑐 = 𝑘𝜎
At least 0% of the realizations lies within σ of μ
At least 75% of the realizations lies within 2σ of μ
At least 89% of the realizations lies within 3σ of μ

Week 2
Moment generating function
If X is a random variable, then the expected value 𝑀A (𝑡) = 𝐸(𝑒 ;A ) is called the Moment
Generating Function (MGF) of X is this expected value exists for all values of t in some
interval of the form |t| < h for some h > 0. It holds that 𝑀A (0) = 1
If X has MGF 𝑀A (𝑡) then 𝑌 = 𝑎𝑋 + 𝑏 has MGF 𝑀B (𝑡) = 𝑒 9; 𝑀A (𝑎𝑡)
MGF and E(X)
(C)
If the MGF of X exists, then 𝐸(𝑋 C ) = 𝑀A (0) for all 𝑟 = 1, 2, … and
?(A ) ); )
𝑀A (𝑡) = 1 + ∑"
C#$ C!
PDF, CDF and MGF
3
PDF à CDF: 𝐹(𝑥) = ∫7" 𝑓(𝑦)𝑑𝑦
E
CDF à PDF: 𝑓(𝑥) = E3 𝐹(𝑥)
"
PDF à MGF: 𝐸(𝑒 ;A ) = ∫7" 𝑒 ;3 𝑓(𝑥)𝑑𝑥
MGF à PDF: recognize

2