Stochastic Modelling - Tutorials & Exam with Solutions

Dublin City University Semester 1 2020/2021
Dr. M. Venker

Exercises to Stochastic Modelling
Sheet 1


Exercise 1 (Markov Chains)
Let the transition matrix of a (time-homogeneous) Markov Chain (Xn )n∈N be given as
 
0.2 0 0.8
P :=  0 0.4 0.6 .
0.9 0.1 0

1. Let the states of the chain be 1,2,3, corresponding to the rows and columns of P . Draw the
transition graph of the Markov Chain.
2. Compute

P (X2 = 3|X0 = 1) .

3. Let the initial distribution q be given by (0.1, 0.5, 0.4). Compute the probability

P (X1 = 2, X3 = 1) .

(Hint: Use Lemmas 1.5 and 1.11.)

Exercise 2 (Markov Chains with independent increments)
Let Y1 , Y2 , . . . be a sequence of independent random variables on the same probability space with
values in Z. Define
n
X
Xn := Yj , n ∈ N.
j=1

Prove that (Xn )n∈N is a Markov Chain by directly validating the Markov property.

(Remark: This process is called a (general) random walk. For Yj taking values ±1 with probabilities
1/2, we recover Example 2.4 (2) from the lecture.)

Exercise 3 (Conditioning on a random variable)
1. Let X, Y be random variables on a common probability space with values in a common discrete
space S. Prove for any i ∈ S
X
P (X = j) = P (Y = i)P (X = j|Y = i) .
i∈S:
P (Y =i)>0


2. Conclude from part 1: If X, Y, Z are random variables on a common probability space with
values in a common discrete space S, then for any i, j ∈ S with P (Y = i) > 0
X
P (X = j|Y = i) = P (Z = k|Y = i)P (X = j|Y = i, Z = k) .
k∈S:
P (Y =i,Z=k)>0

,
,
,