Solution Manual for Stochastic Processes With R An Introduction 1st edition by Olga Korosteleva

ALL 9 CHAPTER COVERED




SOLUTIONS MANUAL

, TABLE OF CONTENTS
CHAPTER 1 ……………………………………………………………………………………. 3
CHAPTER 2 ……………………………………………………………………………………. 31
CHAPTER 3 ……………………………………………………………………………………. 41
CHAPTER 4 ……………………………………………………………………………………. 48
CHAPTER 5 ……………………………………………………………………………………. 60
CHAPTER 6 ……………………………………………………………………………………. 67
CHAPTER 7 ……………………………………………………………………………………. 74
CHAPTER 8 ……………………………………………………………………………………. 81
CHAPTER 9 ……………………………………………………………………………………. 87




2

, CHAPTER 1
0.3 0 . 4 0 . 3
EXERCISE 1.1. For a Markov chain with a one-step transition probability matrix �0.2 0.3 0.5 �
0.8 0 . 1 0 . 1
We compute:

(a) 𝑃𝑃 (𝑋𝑋3 = 2 |𝑋𝑋0 = 1, 𝑋𝑋1 = 2, 𝑋𝑋2 = 3) = 𝑃𝑃 (𝑋𝑋3 = 2 | 𝑋𝑋2 = 3) (by the Markov property)
= 𝑃𝑃32 = 0.1.
(b) 𝑃𝑃 (𝑋𝑋4 = 3 |𝑋𝑋0 = 2, 𝑋𝑋3 = 1) = 𝑃𝑃 (𝑋𝑋4 = 3 | 𝑋𝑋3 = 1) (by the Markov property)
= 𝑃𝑃13 = 0.3.
(c) 𝑃𝑃 (𝑋𝑋0 = 1, 𝑋𝑋1 = 2, 𝑋𝑋2 = 3, 𝑋𝑋3 = 1) = 𝑃𝑃 (𝑋𝑋3 = 1 | 𝑋𝑋0 = 1, 𝑋𝑋1 = 2, 𝑋𝑋2 = 3) 𝑃𝑃 (𝑋𝑋2 = 3 |𝑋𝑋0 = 1,
𝑋𝑋1 = 2) 𝑃𝑃 (𝑋𝑋1 = 2 | 𝑋𝑋0 = 1) 𝑃𝑃 (𝑋𝑋0 = 1) (by conditioning)
= 𝑃𝑃 (𝑋𝑋3 = 1 | 𝑋𝑋2 = 3) 𝑃𝑃 (𝑋𝑋2 = 3 | 𝑋𝑋1 = 2) 𝑃𝑃 (𝑋𝑋1 = 2 | 𝑋𝑋0 = 1) 𝑃𝑃 (𝑋𝑋0 = 1) (by the Markov property)

= 𝑃𝑃31 𝑃𝑃23 𝑃𝑃12 𝑃𝑃 (𝑋𝑋0 = 1) = (0.8) (0.5) (0.4) (1) = 0.16.
(d) We first compute the two-step transition probability matrix. We obtain

0.3 0 . 4 0 . 3
0.3 0 . 4 0.41 0.27 0.32
0.3
𝐏𝐏 = �
(2)
0.2 0.3 0.5 ��0.2 0.3 0.5 �= � 0.52 0.22 0.26�
.

Now we write 0.8 0 . 1 0.8 0 . 1 0.34 0.36 0.30

0.1 0.1
𝑃𝑃 (𝑋𝑋0 = 1, 𝑋𝑋1 = 2, 𝑋𝑋3 = 3, 𝑋𝑋5 = 1) = 𝑃𝑃 (𝑋𝑋5 = 1 | 𝑋𝑋0 = 1, 𝑋𝑋1 = 2, 𝑋𝑋3 = 3) 𝑃𝑃 (𝑋𝑋3 = 3 |𝑋𝑋0 = 1,
𝑋𝑋1 = 2) 𝑃𝑃 (𝑋𝑋1 = 2 | 𝑋𝑋0 = 1) 𝑃𝑃 (𝑋𝑋0 = 1) (by conditioning)
= 𝑃𝑃 (𝑋𝑋5 = 1 | 𝑋𝑋3 = 3) 𝑃𝑃 (𝑋𝑋3 = 3 | 𝑋𝑋1 = 2) 𝑃𝑃 (𝑋𝑋1 = 2 | 𝑋𝑋0 = 1) 𝑃𝑃 (𝑋𝑋0 = 1) (by the Markov property)
(2) (2) 𝑃𝑃(𝑋𝑋 = 1) = (0.34) (0.26) (0.4) (1) = 0.03536.
𝑃𝑃

= 𝑃𝑃31 𝑃𝑃23 12 0

EXERCISE 1.2. (a) We plot a diagram of the Markov chain.

#specifying transition probability matrix
Tm<- matrix(c (1, 0, 0, 0, 0, 0.5, 0, 0, 0, 0.5, 0.2, 0, 0, 0, 0.8,
0, 0, 1, 0, 0, 0, 0, 0, 1, 0), now=5, ndo=5, brow=TRUE)

#transposing transition probability matrix
tm.tr<- t(tm)

#plotting diagram library
(diagram)
Plot mat (tm.tr, arr.length=0.25, arr.width=0.1, box.col="light blue",
box.lwd=1, box. Prop=0.5, box. Size=0.12, box. Type="circle", cex.txt=0.8,
lewd=1, self.cex=0.3, self.shiftx=0.01, self. Shifty=0.09)




3

,State 2 is reflective. The chain leaves that state in one step. Therefore, it forms a separate transient
class that has an infinite period.

Finally, states 3, 4, and 5 communicate and thus belong to the same class. The chain can return to
either state in this class in 3, 6, 9, etc. steps, thus the period is equal to 3. Since there is a positive
probability to leave this class, it is transient.

The R output supports these findings.

#creating Markov chain object library
(markovchain)
Mc<- new ("markovchain", transition Matrix=tm, states=c ("1", "2", "3", "4", "5"))

#computing Markov chain characteristics recurrent
Classes (mc)

"1"

Transient Classes (mc)

"2"
"3" "4" "5"

Absorbing States (mc)

"1"

(c) Below we simulate three trajectories of the chain that start at a randomly chosen state.
4

,#specifying total number of steps
steps<- 25

#specifying seed set. Seed
(4955145)

#specifying initial probability
p0<- c (0.2, 0.2, 0.2, 0.2,
0.2)

#specifying matrix containing states
MC.states<- matrix (NA, now=steps, ndo=3)

#simulating states
for (I in 1:3) {
state0<- state0<- sample (1:5, 1, probe=p0)
MC.states [, I] <- rmarkovchain (n=nsteps-1, object=mc, t0=state0,
include.t0=TRUE)
}

#plotting simulated trajectories
Mat plot (MC.states, type="l", lay=1, lewd=2, col=2:4, at="n", slim=c (1, 5),
lab="Step", lab="State", panel. First=grid ())

Axis (side=1, at=c (1, 5, 10, 15, 20, 25))

points(1:nsteps, MC.states[,1], picha=16,
col=2) points(1:nsteps, MC.states[,2],
picha=16, col=3) points(1:nsteps,
MC.states[,3], picha=16, col=4)




Since state 1 is an absorbing state, sooner or later, the trajectories transition into this state and don’t
leave it.


(d) To find the steady-state probabilities, we need to solve the following equations:


5

, 10 0 0 0


(𝜋𝜋1, 𝜋𝜋2, 𝜋𝜋3, 𝜋𝜋4, 𝜋𝜋5) = (𝜋𝜋1, 𝜋𝜋2, 𝜋𝜋3, 𝜋𝜋4, 𝜋𝜋5) ⎡0.2 5 ⎤ , with the additional condition
0.5 00 000 00 .0.8
⎢ 0 0 1 0 0 ⎥
𝜋𝜋01 = 𝜋𝜋1 + 0.5𝜋𝜋2 + 0.2𝜋𝜋3
𝜋𝜋2 = . It has the degenerate solution
that 𝜋𝜋1 + 𝜋𝜋
Written 2 +the
out, 𝜋𝜋3system
+ 𝜋𝜋4 +becomes
𝜋𝜋5 = 1. � ⎣ 0 0 0 1 0 ⎦
𝜋𝜋3 = 𝜋𝜋4 = 𝜋𝜋5 = 0
𝜋𝜋1 + 𝜋𝜋2 + 𝜋𝜋3 + �𝜋4 + 𝜋𝜋5 = 1
𝜋𝜋1 = 1, 𝜋𝜋2 = 𝜋𝜋3 = 𝜋𝜋4 = 𝜋𝜋5 = 0. This solution is expected because state 1 is an absorbing state, and
So the chain ends up spending 100% of the time there. Having a unique stationary distribution, it is an
Ergodic Markov chain.

Using R, we obtain: steady

States (mc)

1 2 3 4 5
1 0 0 0 0



(e) Here we plot the unconditional probabilities at time 𝑛𝑛 against the time.
#specifying total number of steps
Steps<- 70

#specifying matrix containing probabilities
probes<- matrix (NA, now=steps, ndo=5)

#computing probabilities probes
[1,] <- p0
For (n in 2: nsteps)
Probes [n,] <- probes [n-1,] %*%tm

#plotting probabilities vs. step by state
Mat plot (probes, type="l", lay=1, lewd=2, col=1:5, slim=c (-0.1,
1.1), lab="Step", lab="Probability", panel. First=grid ())

legend("right", c("State 1", "State 2", "State 3", "State4", "State5"), lay=1,
lewd=2, col=1:5)




6

,EXERCISE 1.3. (a) We plot a diagram of the Markov chain.
#specifying transition probability matrix
tm<- matrix(c(0,1,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0.4,0.2,0.2,0.2,
0,0,0,0,0.2,0.4,0.4,0.3,0,0,0.1,0.3,0.1,0.2,0,0,0,0.2,0.2,0.3,0.3,
0, 0, 0, 0.5, 0.2, 0.2, 0.1), now=7, ndo=7, brow=TRUE)

#transposing transition probability matrix
tm.tr<- t(tm)

#plotting diagram library
(diagram)
Plot mat (tm.tr, arr.length=0.3, arr.width=0.1, arroyos=0.58, box.col="light
blue", box.lwd=1, box. Prop=0.5, box. Size=0.09, box. Type="circle", cex.txt=0.8,
lewd=1, self.cex=0.3, self.shiftx=-0.07, self. Shifty=-0.05)




7

,(b) States 1 and 2 form a class and it is recurrent. The period is 2. Once the chain transitions into this
class, it never leaves it and will bounce between the two states.

State 3 is reflecting. The chain leaves this state in one step. This state forms a class of its own. It is a
transient class and its period is infinite.

States 4, 5, 6, and 7 communicate and thus form a class. Its period is one because of the loops.
This class is transient because with positive probability the chain can leave this state and transition
into the {1, 2} class.

From R, we obtain:

#creating Markov chain object library
(markovchain)
Mc<- new ("markovchain", transition Matrix=tm, states=c ("1", "2", "3", "4", "5",
"6", "7"))

#computing Markov chain characteristics recurrent
Classes (mc)

"1" "2"

Transient Classes (mc)

"3"
"4" "5" "6" "7"

Absorbing States (mc)
Character (0)

#creating irreducible Markov chain objects
tm.ir<- matrix(c (0, 1, 1, 0), now=2, ndo=2, brow=TRUE)
mc.ir<-new ("markovchain", transition Matrix=tm.ir, states=c ("1","2"))


8

,#finding periods of irreducible Markov chains
period (mc.ir)

2

(c) Below we simulate two trajectories of the chain that start at a randomly selected state.

#specifying total number of steps
steps<- 25

#specifying seed set. Seed
(3339964)

#specifying initial probability
p0<- c (1/7, 1/7, 1/7, 1/7, 1/7, 1/7, 1/7)

#specifying matrix containing states
MC.states<- matrix (NA, now=steps, ndo=2)

#simulating states
for (I in 1:2) {
state0<- sample (1:7, 1, probe=p0)
MC.states [, I] <- rmarkovchain (n=nsteps-1, object=mc, t0=state0,
include.t0=TRUE)
}

#plotting simulated trajectories
Mat plot (MC.states, type="l", lay=1, lewd=2, col=3:4, slim=c (1, 7), at="n",
lab="Step", lab="State", panel. First=grid ())

Axis (side=1, at=c (1, 5, 10, 15, 20, 25))

Points (1: nsteps, MC.states [, 1], picha=16,
col=3) points (1: nsteps, MC.states [, 2],
picha=16, col=4)




Both simulated trajectories transition to the class {1, 2} sooner or later.

(d) Below we calculate the limiting probabilities.
9

, In R:

#finding steady-state distribution round
(steady States (mc), digits=4)

1 2 3 4 5 6 7
0.5 0.5 0 0 0 0 0

There is a single limiting distribution which means that the chain is ergodic. States 1 and 2 absorb the
chain and then the chain spends 50% of the time in state 1 and the other 50%, in state 2.


(e) Here we plot the unconditional probability vectors 𝑝𝑝𝑛𝑛 against 𝑛𝑛.
#specifying total number of steps
Steps<- 60

#specifying matrix containing probabilities
probes<- matrix (NA, now=steps, ndo=7)

#computing probabilities probes
[1,] <- p0
For (n in 2: nsteps)
Probes [n,] <- probes [n-1,] %*%tm

#plotting probabilities vs. step by state
Mat plot (probes, type="l", lay=1, lewd=2, col=1:7, slim=c (-0.05,
0.6), lab="Step", lab="Probability", panel. First=grid ())

legend("right", c("State 1","State 2","State 3","State 4","State 5","State 6",
"State 7"), lay=1, lewd=2, col=1:7)state 2","state 3","state 4","state
5","state 6", "state 7"), lay=1, col=1:7)




For state 1 and 2 the probabilities converge to 0.5, whereas for all the other states, the probabilities
converge to zero. The curves settle around step 50.


10