Stochastic Processes Notes
October 1, 2025
,Stochastic Processes Notes
Contents
1 Markov Processes 3
2 Classification of States 7
3 Classification of Chains 12
4 Stationary Distributions and the Limit Theorems 15
4.1 Irreducible Markov Chain Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2 General Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5 Reversibility 20
5.1 Interpretation of Detailed Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
6 Handout 2 23
6.1 Exponential Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
6.2 Intuitive Explanation Using Train Arrival Times . . . . . . . . . . . . . . . . . . . . 26
6.3 Generating Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
6.4 Laplace Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
6.5 Random Sums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6.6 Feller’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
7 Chains with Finitely many States 36
7.1 Diagonalizable P matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
7.2 Jacobian P matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
8 Birth Process and the Poisson Process 42
8.1 Poisson Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
8.1.1 Intuitive Proof of Theorem (Poisson Process follows a Poisson Distribution) . 44
8.2 Poisson Process using Exponential Distribution . . . . . . . . . . . . . . . . . . . . . 45
Page 1 of 67
,Stochastic Processes Notes
8.3 Poisson Process and Uniform Distribution . . . . . . . . . . . . . . . . . . . . . . . . 46
8.4 Birth Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
8.4.1 How to Solve Forward System? . . . . . . . . . . . . . . . . . . . . . . . . . . 52
9 Continuous-Time Markov Chains 54
9.1 Generation Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
9.2 Continuous MC given as Discrete MC . . . . . . . . . . . . . . . . . . . . . . . . . . 57
9.3 Stationary Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
10 Handout 4 62
11 Birth-Death Processes 64
Page 2 of 67
, Stochastic Processes Notes
Chapter 1
Markov Processes
Definition 1.0.1. (Markov Chains)
Let S be a discrete set. A Markov Chain is a sequence of random variables X0 , X1 , . . . taking
values in S with the property that:
P(Xn+1 = j | X0 = x0 , . . . , Xn−1 = xn−1 , Xn = i) = P(Xn+1 = j | Xn = i)
for all x0 , . . . , xn−1 , i, j ∈ S, and n ≥ 0. The set S is the state space of the Markov chain.
Definition 1.0.2. (Time-Homogeneous)
A Markov-Chain is time-homogeneous if the probabilities do not depend on n. That is:
P(Xn+1 = j | Xn = i) = P(X1 = j|X0 = i)
Remark: We mean that time has no effect on what happens. Only the state is the one affecting
the system.
Definition 1.0.3. (Transition Matrix)
Let X0 , X1 , . . . be a time-homogeneous Markov Chain. Let S = {1, 2, 3, 4, 5, . . . } be the state space.
Since the probabilities from one state to another are fixed we can summarize them into a matrix
representing transition from one state to another.
1 2 3 ...
1 a11 a12 a13 ...
2 a21 a22 a23 ...
3 a31 a32 a33 ...
.. .. .. ..
. . . .
Page 3 of 67
October 1, 2025
,Stochastic Processes Notes
Contents
1 Markov Processes 3
2 Classification of States 7
3 Classification of Chains 12
4 Stationary Distributions and the Limit Theorems 15
4.1 Irreducible Markov Chain Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2 General Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5 Reversibility 20
5.1 Interpretation of Detailed Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
6 Handout 2 23
6.1 Exponential Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
6.2 Intuitive Explanation Using Train Arrival Times . . . . . . . . . . . . . . . . . . . . 26
6.3 Generating Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
6.4 Laplace Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
6.5 Random Sums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6.6 Feller’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
7 Chains with Finitely many States 36
7.1 Diagonalizable P matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
7.2 Jacobian P matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
8 Birth Process and the Poisson Process 42
8.1 Poisson Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
8.1.1 Intuitive Proof of Theorem (Poisson Process follows a Poisson Distribution) . 44
8.2 Poisson Process using Exponential Distribution . . . . . . . . . . . . . . . . . . . . . 45
Page 1 of 67
,Stochastic Processes Notes
8.3 Poisson Process and Uniform Distribution . . . . . . . . . . . . . . . . . . . . . . . . 46
8.4 Birth Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
8.4.1 How to Solve Forward System? . . . . . . . . . . . . . . . . . . . . . . . . . . 52
9 Continuous-Time Markov Chains 54
9.1 Generation Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
9.2 Continuous MC given as Discrete MC . . . . . . . . . . . . . . . . . . . . . . . . . . 57
9.3 Stationary Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
10 Handout 4 62
11 Birth-Death Processes 64
Page 2 of 67
, Stochastic Processes Notes
Chapter 1
Markov Processes
Definition 1.0.1. (Markov Chains)
Let S be a discrete set. A Markov Chain is a sequence of random variables X0 , X1 , . . . taking
values in S with the property that:
P(Xn+1 = j | X0 = x0 , . . . , Xn−1 = xn−1 , Xn = i) = P(Xn+1 = j | Xn = i)
for all x0 , . . . , xn−1 , i, j ∈ S, and n ≥ 0. The set S is the state space of the Markov chain.
Definition 1.0.2. (Time-Homogeneous)
A Markov-Chain is time-homogeneous if the probabilities do not depend on n. That is:
P(Xn+1 = j | Xn = i) = P(X1 = j|X0 = i)
Remark: We mean that time has no effect on what happens. Only the state is the one affecting
the system.
Definition 1.0.3. (Transition Matrix)
Let X0 , X1 , . . . be a time-homogeneous Markov Chain. Let S = {1, 2, 3, 4, 5, . . . } be the state space.
Since the probabilities from one state to another are fixed we can summarize them into a matrix
representing transition from one state to another.
1 2 3 ...
1 a11 a12 a13 ...
2 a21 a22 a23 ...
3 a31 a32 a33 ...
.. .. .. ..
. . . .
Page 3 of 67
