Machine Learning — Course Notes
MACHINE LEARNING COMPREHENSIVE COURSE NOTES ALGORITHMS, PYTHON
IMPLEMENTATIONS, AND PRACTICE PROBLEMS LATEST VERSION 2026/2027
Machine Learning
Comprehensive Course Notes
Algorithms, Python Implementations, and Practice Problems
Page 1
, Machine Learning — Course Notes
1. What Is Machine Learning?
Machine learning (ML) is the study of algorithms that improve their performance on a
task through experience, typically expressed as data, rather than through explicit hand-
written rules. Formally, following Tom Mitchell's definition: a program learns from
experience E with respect to task T and performance measure P if its performance on T,
as measured by P, improves with E.
ML problems are usually grouped into three broad categories. Supervised learning uses
labeled examples (input-output pairs) to learn a mapping from inputs to outputs; this
covers regression (continuous output) and classification (discrete output). Unsupervised
learning works with unlabeled data to discover structure, such as clusters or lower-
dimensional representations. Reinforcement learning trains an agent to choose actions in
an environment to maximize cumulative reward.
A typical supervised pipeline: collect and clean data, split into training/validation/test
sets, choose a model class and loss function, fit parameters by minimizing loss on the
training set, tune hyperparameters using the validation set, and finally report
performance on the held-out test set. The central tension throughout is the bias-variance
tradeoff: a model that is too simple underfits (high bias), while a model that is too
complex overfits (high variance) and fails to generalize to new data.
A minimal supervised-learning workflow with scikit-learn
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
# X: feature matrix (n_samples, n_features), y: targets
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42)
model = LinearRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)
mse = mean_squared_error(y_test, predictions)
print(f"Test MSE: {mse:.4f}")
2. Linear Regression
Linear regression models the relationship between input features x and a continuous
target y as a linear function: y_hat = w^T x + b, where w is a weight vector and b is a bias
(intercept) term. Learning means choosing w and b that minimize a loss function over the
training data, almost always the mean squared error (MSE): L(w, b) = (1/n) * sum( (y_i -
y_hat_i)^2 ).
Page 2
, Machine Learning — Course Notes
There are two standard ways to fit the parameters. The closed-form (normal equation)
solution sets the gradient of the MSE to zero and solves directly: w = (X^T X)^-1 X^T y.
This is exact but becomes expensive when the number of features is large, because it
requires inverting an (n_features x n_features) matrix. The iterative alternative is gradient
descent, which repeatedly nudges the parameters in the direction that most reduces the
loss: w := w - alpha * dL/dw, where alpha is the learning rate.
Feature scaling (standardizing each feature to zero mean and unit variance) matters a
great deal for gradient descent: features on very different scales cause the loss surface to
become a narrow, elongated valley, which makes gradient descent oscillate and converge
slowly. It does not affect the closed-form solution's correctness, only its numerical
stability.
Gradient descent for linear regression, implemented from scratch
import numpy as np
def fit_linear_regression(X, y, lr=0.01, epochs=1000):
n_samples, n_features = X.shape
w = np.zeros(n_features)
b = 0.0
for epoch in range(epochs):
y_hat = X @ w + b
error = y_hat - y
# Gradients of the MSE loss
dw = (2 / n_samples) * (X.T @ error)
db = (2 / n_samples) * np.sum(error)
w -= lr * dw
b -= lr * db
return w, b
# Usage
w, b = fit_linear_regression(X_train, y_train, lr=0.05, epochs=2000)
y_pred = X_test @ w + b
3. Logistic Regression
Logistic regression is a linear model for binary classification. Rather than predicting y
directly, it predicts the probability that y = 1 by passing a linear combination of features
through the sigmoid function: p = sigma(z) = 1 / (1 + e^-z), where z = w^T x + b. Because
sigma maps any real number to the interval (0, 1), p can be interpreted as a probability.
The parameters are fit by minimizing the binary cross-entropy (log loss): L = -(1/n) *
sum( y_i * log(p_i) + (1 - y_i) * log(1 - p_i) ). This loss is convex in w and b, so gradient
descent (or Newton's method) reliably finds the global minimum. At prediction time, a
Page 3
MACHINE LEARNING COMPREHENSIVE COURSE NOTES ALGORITHMS, PYTHON
IMPLEMENTATIONS, AND PRACTICE PROBLEMS LATEST VERSION 2026/2027
Machine Learning
Comprehensive Course Notes
Algorithms, Python Implementations, and Practice Problems
Page 1
, Machine Learning — Course Notes
1. What Is Machine Learning?
Machine learning (ML) is the study of algorithms that improve their performance on a
task through experience, typically expressed as data, rather than through explicit hand-
written rules. Formally, following Tom Mitchell's definition: a program learns from
experience E with respect to task T and performance measure P if its performance on T,
as measured by P, improves with E.
ML problems are usually grouped into three broad categories. Supervised learning uses
labeled examples (input-output pairs) to learn a mapping from inputs to outputs; this
covers regression (continuous output) and classification (discrete output). Unsupervised
learning works with unlabeled data to discover structure, such as clusters or lower-
dimensional representations. Reinforcement learning trains an agent to choose actions in
an environment to maximize cumulative reward.
A typical supervised pipeline: collect and clean data, split into training/validation/test
sets, choose a model class and loss function, fit parameters by minimizing loss on the
training set, tune hyperparameters using the validation set, and finally report
performance on the held-out test set. The central tension throughout is the bias-variance
tradeoff: a model that is too simple underfits (high bias), while a model that is too
complex overfits (high variance) and fails to generalize to new data.
A minimal supervised-learning workflow with scikit-learn
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
# X: feature matrix (n_samples, n_features), y: targets
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42)
model = LinearRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)
mse = mean_squared_error(y_test, predictions)
print(f"Test MSE: {mse:.4f}")
2. Linear Regression
Linear regression models the relationship between input features x and a continuous
target y as a linear function: y_hat = w^T x + b, where w is a weight vector and b is a bias
(intercept) term. Learning means choosing w and b that minimize a loss function over the
training data, almost always the mean squared error (MSE): L(w, b) = (1/n) * sum( (y_i -
y_hat_i)^2 ).
Page 2
, Machine Learning — Course Notes
There are two standard ways to fit the parameters. The closed-form (normal equation)
solution sets the gradient of the MSE to zero and solves directly: w = (X^T X)^-1 X^T y.
This is exact but becomes expensive when the number of features is large, because it
requires inverting an (n_features x n_features) matrix. The iterative alternative is gradient
descent, which repeatedly nudges the parameters in the direction that most reduces the
loss: w := w - alpha * dL/dw, where alpha is the learning rate.
Feature scaling (standardizing each feature to zero mean and unit variance) matters a
great deal for gradient descent: features on very different scales cause the loss surface to
become a narrow, elongated valley, which makes gradient descent oscillate and converge
slowly. It does not affect the closed-form solution's correctness, only its numerical
stability.
Gradient descent for linear regression, implemented from scratch
import numpy as np
def fit_linear_regression(X, y, lr=0.01, epochs=1000):
n_samples, n_features = X.shape
w = np.zeros(n_features)
b = 0.0
for epoch in range(epochs):
y_hat = X @ w + b
error = y_hat - y
# Gradients of the MSE loss
dw = (2 / n_samples) * (X.T @ error)
db = (2 / n_samples) * np.sum(error)
w -= lr * dw
b -= lr * db
return w, b
# Usage
w, b = fit_linear_regression(X_train, y_train, lr=0.05, epochs=2000)
y_pred = X_test @ w + b
3. Logistic Regression
Logistic regression is a linear model for binary classification. Rather than predicting y
directly, it predicts the probability that y = 1 by passing a linear combination of features
through the sigmoid function: p = sigma(z) = 1 / (1 + e^-z), where z = w^T x + b. Because
sigma maps any real number to the interval (0, 1), p can be interpreted as a probability.
The parameters are fit by minimizing the binary cross-entropy (log loss): L = -(1/n) *
sum( y_i * log(p_i) + (1 - y_i) * log(1 - p_i) ). This loss is convex in w and b, so gradient
descent (or Newton's method) reliably finds the global minimum. At prediction time, a
Page 3
