CS 7641 Final Exam Review | Machine Learning Study Notes 2026/2027

CS 7641 Final Exam Review | Machine
Learning Study Notes 2026/2027


Four optimization approaches - correct answer-1) Generate and test
2) Calculus
3) Newton's Method
4) Randomized Optimization


Hill Climbing Algorithm - correct answer-Guess x∈X
Repeat the following:
Let n*=argmax_n∈N(x) f(n)
If f(n)>f(x): x=n
Else: stop


Disadvantage:
- Get stuck in local optima


Randomized Restart Hill Climbing - correct answer-Same as Hill Climbing but once
local optimum reached, restart again with a different starting x


Advantage:
- Won't get stuck in local optimum
- Not much more expensive than HC (constant factor)

, Disadvantage:
- May not do better than enumeration (depends on size of attraction basin
around global optimum)


Entropy - correct answer--∑p(s)log₂p(s)


Number of bits per symbol (probability of symbol X # of bits to describe that
symbol)


Joint Entropy - correct answer-H(x,y)=-∑p(x,y)log₂p(x,y)


Randomness contained in two variables together


Conditional Entropy - correct answer-H(y|x)=-∑p(x,y)log₂p(y|x)


Randomness of one variable given the other variable


Entropy if x and y are independent - correct answer-H(Y|X)=H(Y) Y doesn't get any
info from x
H(X,Y)=H(X)+H(Y) Joint entropy is sum


Mutual Information - correct answer-I(x,y)=H(y)-H(y|x)=I(y,x)


Measure of reduction of randomness of variable given some knowledge of
another variable.