Machine Learning
Cambiago Silvia Academic Year 2025-2026 Prof. Tim Landgraf
WHAT IS MACHINE LEARNING
Machine Learning is a subfield of Artificial Intelligence, which is the broad field of making
computers perform tasks that normally require human intelligence, such as reasoning, planning,
understanding language, or recognizing images.
Machine Learning is focused on building
systems that can learn from data rather than
being explicitly programmed. ML is often
called statistical learning or sub-symbolic AI,
because it relies on statistics and numeric
patterns rather than human-readable logic.
Inside ML there is Deep Learning, another subset that uses artificial neural networks with many
layers to learn complex representations from large datasets. For example, deep learning powers
image recognition and speech understanding.
LEARNING TYPES
Machine Learning methods can be broadly categorized according to the type of data and
feedback available during training. The most fundamental distinction is between supervised
and unsupervised learning:
F Supervised: the algorithm is trained on a dataset consisting of input-output pairs. The
inputs are typically represented as vectors 𝑋 = {𝑥! , 𝑥" , … 𝑥# } belonging to some feature
space ℝ$ , while the associated outputs, or labels, are 𝑌 = {𝑦! , 𝑦" , … , 𝑦# } in ℝ% ,so there
are examples of the correct answers. The objective is to learn a function f that maps
each input 𝑥& to its corresponding label 𝑦& , and, given a new input, can accurately
predict the corresponding output;
F Unsupervised: data comes without labels or outputs, and the goal is rather to discover
hidden patterns, groupings, or structures within the data itself. Typical tasks include
clustering, where the algorithm groups similar data points together, and dimensionality
reduction, where it identifies the most informative directions in which the data vary.
Between these two extremes lie semi-supervised and self-supervised learning. In semi-
supervised learning, a small amount of labeled data is combined with a larger amount of
unlabeled data, taking advantage of the latter to improve performance without the costly
process of manual labeling. On the other hand, self-supervised learning, which has become
increasingly popular in recent years, uses the data itself to create pseudo-labels or internal
learning objectives (like predicting missing parts of an image or the next word in a sentence).
This approach has enabled the training of extremely large models without human annotation.
1
,NOTATION
At its core, a machine learning model can be viewed as a mathematical function that transforms
inputs into outputs. Inputs are denoted as 𝑥, the model as 𝑓, and the predicted output as 𝑦,. The
model f applies a linear transformation to x, producing a prediction 𝑦, = 𝑓(𝑥).
When there are several observations, they are arranged into a matrix X, where each row
corresponds to one sample and each column to one feature. Thus, if there are N samples and
M features, X has size N ´ M. The true outputs (or labels) form a vector Y, and the model
attempts to approximate the relationship 𝑌 ≈ 𝑓(𝑋). In practice, predictions are not perfect, so
the relationship becomes 𝑦, = 𝑓(𝑥) = 𝑦 + ϵ, where ϵ represents the error term. The task of
learning consists of finding the function f that minimizes this error over all training samples.
DECISION BOUNDARY
In classification tasks, the goal of the learning algorithm is to find a decision boundary, so a
line (or, in higher dimensions, a surface) that best separates the two groups. At first, only the
training data is available. Using this data, the algorithm searches for a pattern, adjusting its
internal parameters until it finds the boundary that minimizes classification errors. This process
is guided by an optimization objective, which defines what “best” means, typically minimizing
the number of misclassified points.
In the diagram in the figure, the two classes are the fishes sea bass
and salmon, and the solid black line represents this learned decision
boundary. The dotted lines around it can be thought of as uncertainty
margins: the model’s prediction becomes less certain the closer a fish
lies to the boundary.
Once the decision boundary has been learned, the model is ready to classify new data. Given a
new fish whose class is unknown, its features are measured (weight and lightness in this case)
and this fish is plotted as a new point in the graph. Depending on which side of the decision
boundary it falls, the model assigns it a label: if it’s on the side where most salmon lie, it
predicts “salmon”, if on the opposite side, “sea bass”. The decision boundary represents the
model’s internal rule for separating classes, but the precise position of this boundary depends
on how the model is trained and on the cost function used.
A cost function measures how wrong the model’s predictions are. By minimizing this cost
during training, the algorithm adjusts its boundary until it reaches the optimal position.
2
,In two dimensions, the decision boundary can take different forms depending on the
complexity of the model:
F A simple linear model (left figure) produces a straight boundary. This corresponds to
algorithms like logistic regression or linear discriminant analysis, which assume a
roughly linear relationship between features and classes;
F A more flexible model (right figure) can learn nonlinear boundaries, bending around
the data points to capture complex patterns. Neural networks or decision trees, for
example, can adapt their boundaries in highly irregular shapes to fit the data better.
An essential challenge in machine learning is to achieve generalization, that is, to perform well
not only on the data the model has seen during training but also on new, unseen data. On the
other hand, when a model learns the training data too well, including its noise or random
fluctuations, another phenomenon called overfitting occurs. Models that overfit will perfectly
classify every training point, but will fail when faced with new data. The two figures contrast
these ideas. The left-hand image, with a simple linear boundary, represents a model that
generalizes well: it may misclassify a few training points but captures the overall trend. The
right-hand image shows a highly complex boundary, that corresponds to an overfitted model
that performs perfectly on training data but poorly on unseen examples. In practice, the art of
machine learning lies in finding the right balance between bias (model too simple) and variance
(model too complex). A good model generalizes, while a bad one memorizes.
K-NEAREST NEIGHBOR CLASSIFICATION
An algorithm that solves the classification problem is the k-Nearest Neighbor (k-NN). Its
underlying assumption is that similar objects are likely to belong to the same class. The key
idea is proximity: points that are close in the feature space (with reference to the fish example,
similar in weight and lightness) are assumed to be of the same category. The k-NN algorithm
can be described in two main steps:
F Training: stores all the training data, so all N pairs (𝑥& , 𝑦& ), with 0 ≤ 𝑖 ≤ 𝑁, where 𝑥&
represents the features of a fish and 𝑦& its label. For binary classification, 𝑦& ∈ {0,1};
F Classification: when a new sample x is given, the algorithm:
o Defines the k neighborhood of x, so the k closest points to x, which form the set
𝑁% (𝑥);
!
o Computes the result 𝑦, = % ∑'! ∈)" (') 𝑦& , which corresponds to the average of the
labels of the nearest neighbors.
The choice of k, the number of neighbors, is critical because it directly influences the
smoothness of the decision boundary. If k is small, the boundary will be irregular and will
perfectly fit the training data. However, such a model is very sensitive to noise: one mislabeled
or outlying sample can lead to incorrect predictions. If k is large, the decision boundary
becomes smoother. The model considers more neighbors, which reduces the influence of noise
but also risks oversimplifying the problem. When k is too high, the boundary may become so
smooth that it can no longer capture the real distinctions between the classes (underfitting).
3
, This method is conceptually very simple and easy to implement. It works for both classification
(predicting categories) and regression (predicting continuous values), and has only one
hyperparameter k, making it easy to tune. Additionally, it is non-parametric, meaning it makes
no assumptions about the shape of the underlying data distribution. This flexibility allows it to
adapt to complex, nonlinear patterns. On the other hand, since all data must be stored and
compared during prediction, it can be computationally expensive for large datasets. It also
suffers from the curse of dimensionality: in high-dimensional spaces, distances between points
become less meaningful, and performance degrades significantly. Finally, choosing the right
value of k is not always straightforward.
K-NN (ADVANCED)
In classification tasks, the target variable 𝑦& represents a category. When predicting the label
of a new sample x, the k closest neighbors are considered, so the k training points that are most
similar to x in terms of distance. The algorithm then chooses the class that occurs most
frequently among these neighbors. Formally, this can be expressed as:
𝑓:(𝑥) = 𝑎𝑟𝑔 𝑚𝑎𝑥,∈𝒞 ? 1(𝑦& = 𝑐)
&∈)" (')
where 1(𝑦& = 𝑐) is the indicator function, that equals 1 if the i-th neighbor belongs to class c,
0 otherwise.
In regression tasks, the target variable 𝑦& is real-valued. The prediction is the average (mean)
of the k nearest neighbors’ target values:
1
𝑓:(𝑥) = ? 𝑦&
𝐾
&∈)" (')
This means that, instead of counting votes, the arithmetic mean of the neighbors’ outputs is
taken. For example, to estimate the price of a new house based on its size and number of
bedrooms, find the three most similar houses in the dataset and predict the price as the average
of those three. Predictions can be improved by giving more weight to closer neighbors, since
nearby points may influence the prediction more strongly than distant ones. This leads to a
distance-weighted version of the formula:
∑&∈)" (') 𝑤& 𝑦&
𝑓:(𝑥) =
∑&∈)" (') 𝑤&
where each weight is defined as:
1
𝑤& =
𝑑(𝑥, 𝑥& ) + ε
where 𝑑(𝑥, 𝑥& ) is the distance between x and its neighbor 𝑥& , and ε is a small constant to prevent
division by 0. In this way, the closest points contribute more heavily to the prediction. This
algorithm’s hyperparameters (parameters defined before training the ML model, in contrast to
parameters, characteristics that the model learns from the data) are:
F K: number of neighbors, usually set as an odd number to avoid ties in classification;
F Distance metric: commonly Euclidean norm (L2-norm), Manhattan norm (L1-norm) or
a general Lp-norm.
4
Cambiago Silvia Academic Year 2025-2026 Prof. Tim Landgraf
WHAT IS MACHINE LEARNING
Machine Learning is a subfield of Artificial Intelligence, which is the broad field of making
computers perform tasks that normally require human intelligence, such as reasoning, planning,
understanding language, or recognizing images.
Machine Learning is focused on building
systems that can learn from data rather than
being explicitly programmed. ML is often
called statistical learning or sub-symbolic AI,
because it relies on statistics and numeric
patterns rather than human-readable logic.
Inside ML there is Deep Learning, another subset that uses artificial neural networks with many
layers to learn complex representations from large datasets. For example, deep learning powers
image recognition and speech understanding.
LEARNING TYPES
Machine Learning methods can be broadly categorized according to the type of data and
feedback available during training. The most fundamental distinction is between supervised
and unsupervised learning:
F Supervised: the algorithm is trained on a dataset consisting of input-output pairs. The
inputs are typically represented as vectors 𝑋 = {𝑥! , 𝑥" , … 𝑥# } belonging to some feature
space ℝ$ , while the associated outputs, or labels, are 𝑌 = {𝑦! , 𝑦" , … , 𝑦# } in ℝ% ,so there
are examples of the correct answers. The objective is to learn a function f that maps
each input 𝑥& to its corresponding label 𝑦& , and, given a new input, can accurately
predict the corresponding output;
F Unsupervised: data comes without labels or outputs, and the goal is rather to discover
hidden patterns, groupings, or structures within the data itself. Typical tasks include
clustering, where the algorithm groups similar data points together, and dimensionality
reduction, where it identifies the most informative directions in which the data vary.
Between these two extremes lie semi-supervised and self-supervised learning. In semi-
supervised learning, a small amount of labeled data is combined with a larger amount of
unlabeled data, taking advantage of the latter to improve performance without the costly
process of manual labeling. On the other hand, self-supervised learning, which has become
increasingly popular in recent years, uses the data itself to create pseudo-labels or internal
learning objectives (like predicting missing parts of an image or the next word in a sentence).
This approach has enabled the training of extremely large models without human annotation.
1
,NOTATION
At its core, a machine learning model can be viewed as a mathematical function that transforms
inputs into outputs. Inputs are denoted as 𝑥, the model as 𝑓, and the predicted output as 𝑦,. The
model f applies a linear transformation to x, producing a prediction 𝑦, = 𝑓(𝑥).
When there are several observations, they are arranged into a matrix X, where each row
corresponds to one sample and each column to one feature. Thus, if there are N samples and
M features, X has size N ´ M. The true outputs (or labels) form a vector Y, and the model
attempts to approximate the relationship 𝑌 ≈ 𝑓(𝑋). In practice, predictions are not perfect, so
the relationship becomes 𝑦, = 𝑓(𝑥) = 𝑦 + ϵ, where ϵ represents the error term. The task of
learning consists of finding the function f that minimizes this error over all training samples.
DECISION BOUNDARY
In classification tasks, the goal of the learning algorithm is to find a decision boundary, so a
line (or, in higher dimensions, a surface) that best separates the two groups. At first, only the
training data is available. Using this data, the algorithm searches for a pattern, adjusting its
internal parameters until it finds the boundary that minimizes classification errors. This process
is guided by an optimization objective, which defines what “best” means, typically minimizing
the number of misclassified points.
In the diagram in the figure, the two classes are the fishes sea bass
and salmon, and the solid black line represents this learned decision
boundary. The dotted lines around it can be thought of as uncertainty
margins: the model’s prediction becomes less certain the closer a fish
lies to the boundary.
Once the decision boundary has been learned, the model is ready to classify new data. Given a
new fish whose class is unknown, its features are measured (weight and lightness in this case)
and this fish is plotted as a new point in the graph. Depending on which side of the decision
boundary it falls, the model assigns it a label: if it’s on the side where most salmon lie, it
predicts “salmon”, if on the opposite side, “sea bass”. The decision boundary represents the
model’s internal rule for separating classes, but the precise position of this boundary depends
on how the model is trained and on the cost function used.
A cost function measures how wrong the model’s predictions are. By minimizing this cost
during training, the algorithm adjusts its boundary until it reaches the optimal position.
2
,In two dimensions, the decision boundary can take different forms depending on the
complexity of the model:
F A simple linear model (left figure) produces a straight boundary. This corresponds to
algorithms like logistic regression or linear discriminant analysis, which assume a
roughly linear relationship between features and classes;
F A more flexible model (right figure) can learn nonlinear boundaries, bending around
the data points to capture complex patterns. Neural networks or decision trees, for
example, can adapt their boundaries in highly irregular shapes to fit the data better.
An essential challenge in machine learning is to achieve generalization, that is, to perform well
not only on the data the model has seen during training but also on new, unseen data. On the
other hand, when a model learns the training data too well, including its noise or random
fluctuations, another phenomenon called overfitting occurs. Models that overfit will perfectly
classify every training point, but will fail when faced with new data. The two figures contrast
these ideas. The left-hand image, with a simple linear boundary, represents a model that
generalizes well: it may misclassify a few training points but captures the overall trend. The
right-hand image shows a highly complex boundary, that corresponds to an overfitted model
that performs perfectly on training data but poorly on unseen examples. In practice, the art of
machine learning lies in finding the right balance between bias (model too simple) and variance
(model too complex). A good model generalizes, while a bad one memorizes.
K-NEAREST NEIGHBOR CLASSIFICATION
An algorithm that solves the classification problem is the k-Nearest Neighbor (k-NN). Its
underlying assumption is that similar objects are likely to belong to the same class. The key
idea is proximity: points that are close in the feature space (with reference to the fish example,
similar in weight and lightness) are assumed to be of the same category. The k-NN algorithm
can be described in two main steps:
F Training: stores all the training data, so all N pairs (𝑥& , 𝑦& ), with 0 ≤ 𝑖 ≤ 𝑁, where 𝑥&
represents the features of a fish and 𝑦& its label. For binary classification, 𝑦& ∈ {0,1};
F Classification: when a new sample x is given, the algorithm:
o Defines the k neighborhood of x, so the k closest points to x, which form the set
𝑁% (𝑥);
!
o Computes the result 𝑦, = % ∑'! ∈)" (') 𝑦& , which corresponds to the average of the
labels of the nearest neighbors.
The choice of k, the number of neighbors, is critical because it directly influences the
smoothness of the decision boundary. If k is small, the boundary will be irregular and will
perfectly fit the training data. However, such a model is very sensitive to noise: one mislabeled
or outlying sample can lead to incorrect predictions. If k is large, the decision boundary
becomes smoother. The model considers more neighbors, which reduces the influence of noise
but also risks oversimplifying the problem. When k is too high, the boundary may become so
smooth that it can no longer capture the real distinctions between the classes (underfitting).
3
, This method is conceptually very simple and easy to implement. It works for both classification
(predicting categories) and regression (predicting continuous values), and has only one
hyperparameter k, making it easy to tune. Additionally, it is non-parametric, meaning it makes
no assumptions about the shape of the underlying data distribution. This flexibility allows it to
adapt to complex, nonlinear patterns. On the other hand, since all data must be stored and
compared during prediction, it can be computationally expensive for large datasets. It also
suffers from the curse of dimensionality: in high-dimensional spaces, distances between points
become less meaningful, and performance degrades significantly. Finally, choosing the right
value of k is not always straightforward.
K-NN (ADVANCED)
In classification tasks, the target variable 𝑦& represents a category. When predicting the label
of a new sample x, the k closest neighbors are considered, so the k training points that are most
similar to x in terms of distance. The algorithm then chooses the class that occurs most
frequently among these neighbors. Formally, this can be expressed as:
𝑓:(𝑥) = 𝑎𝑟𝑔 𝑚𝑎𝑥,∈𝒞 ? 1(𝑦& = 𝑐)
&∈)" (')
where 1(𝑦& = 𝑐) is the indicator function, that equals 1 if the i-th neighbor belongs to class c,
0 otherwise.
In regression tasks, the target variable 𝑦& is real-valued. The prediction is the average (mean)
of the k nearest neighbors’ target values:
1
𝑓:(𝑥) = ? 𝑦&
𝐾
&∈)" (')
This means that, instead of counting votes, the arithmetic mean of the neighbors’ outputs is
taken. For example, to estimate the price of a new house based on its size and number of
bedrooms, find the three most similar houses in the dataset and predict the price as the average
of those three. Predictions can be improved by giving more weight to closer neighbors, since
nearby points may influence the prediction more strongly than distant ones. This leads to a
distance-weighted version of the formula:
∑&∈)" (') 𝑤& 𝑦&
𝑓:(𝑥) =
∑&∈)" (') 𝑤&
where each weight is defined as:
1
𝑤& =
𝑑(𝑥, 𝑥& ) + ε
where 𝑑(𝑥, 𝑥& ) is the distance between x and its neighbor 𝑥& , and ε is a small constant to prevent
division by 0. In this way, the closest points contribute more heavily to the prediction. This
algorithm’s hyperparameters (parameters defined before training the ML model, in contrast to
parameters, characteristics that the model learns from the data) are:
F K: number of neighbors, usually set as an odd number to avoid ties in classification;
F Distance metric: commonly Euclidean norm (L2-norm), Manhattan norm (L1-norm) or
a general Lp-norm.
4
