Deep Learning: learning the features of data and simplifying data
representations for the purpose of finding patterns.
Perceptron
● A perceptron is an artificial neuron
● It is the simplest possible Neural Network
● The initial perceptron design involved processing multiple binary inputs by multiplying
each input with a corresponding weight, summing the results along with a bias
parameter to yield a binary output (0 or 1).
● Perceptrons are linear functions:
○ Too simple and abstract to link the outputs of one perceptron to the inputs of
the next in a large network.
○ Composing together linear functions will only ever give you another linear
function. We’re not creating models that can learn non-linear functions.
Activation function
● An activation function is a scalar function (a function from a number to another
number) which we apply to the output of a perceptron after all the weighted inputs
have been combined.
● Typically, the activation function is nonlinear, which allows the network to learn
complex patterns in data.
● Not using an activation function is also called using a linear activation (linear
regression model).
Activation function Plot Goal
Linear activation Also known as the "no activation" function"
function (multiplied x1.0), where the activation is
proportional to the input. The function doesn't
do anything to the weighted sum of the input, it
simply spits out the value it was given.
, Binary Depends on a threshold value that decides
Non-linear step whether a neuron should be activated or not.
Sigmoid Converts input into a probability between 0 and
1; often used in binary classification.
Softmax Converts input into categorical probability
distribution by compressing multiple inputs into
a range between 0 and 1; used in multi-class
classification.
Tanh Similar to sigmoid but ranges from -1 to 1;
useful in hidden layers of a neural network.
ReLU Sets every negative input to zero, and keeps
everything else the same. Allows only positive
values to pass through; helps with non-linear
complex mappings and vanishing gradient
problems.
Leaky Similar to ReLU but allows a small, non-zero
ReLU gradient when the unit is not active; helps
prevent dead neurons.
Neural network
● Any arrangement of perceptrons and nonlinearities.
,Multilayer perceptron (MLP)
● A fully connected feed-forward artificial neural network with at least three layers
(input, output, and at least one hidden layer)
● Consists of:
○ Layer of hidden units in the middle, each of which acts as a perceptron
with a non-linearity (activation function), connecting to all input nodes.
○ One or more output nodes, connecting to all nodes in the hidden layer.
● Features:
1. There are no cycles, the network “feeds forward” from input to output.
2. Nodes in the same layer are not connected to each other, or to any other
layer than the previous one.
3. Each layer is fully connected to the previous layer, every node in one layer
connects to every node in the layer before it.
● Note: Every orange and blue line in this picture represents one parameter of
the model.
● Although perceptrons can only understand linear relationships between the input and
output data provided, MLPs are a neural network where the mapping between inputs
and output is non-linear.
● It learns weights in the process of backpropagation.
Finding good weights
Loss function
● Begin by determining a loss function to assess model performance.
● Seek weights (model parameters) that result in minimal loss over the data.
● Lower loss corresponds to a better-performing model.
● Note: It’s nice if the loss is zero when the prediction is perfect, but this isn’t required.
● Can be defined for a single instance or for all instances in the data. Usually, the loss
over the whole data is just the average loss over all instances.
machine learning loss function
algorithm
regression squared distance between y normal distribution
errors and t (fixed variance)
absolute take the absolute laplace distribution
errors magnitude of the (fixed variance)
difference between
, network output and the
target value
classification log loss binary cross-entropy Bernoulli distribution
log loss cross-entropy Categorical
distribution
hinge loss <none>
Loss Landscape
● Model space consists of all possible models, forming a plane.
● Loss can be visualized as a surface above this plane.
● Every point on the plane corresponds to a specific set of
weights (θ), and the loss function defines the loss for each
point.
Optimization Goal
● Objective is to navigate the loss surface and find a set of
weights (θ) that minimizes the loss on the chosen dataset:
● Use calculus to identify the lowest point on the loss surface.
● In one dimension, derivatives indicate the slope of a tangent
line, representing how much the function rises.
● For multiple dimensions, compute partial derivatives (derivatives with respect to
one of those variables, with the others held constant), forming the gradient.
● The gradient is a vector pointing in the direction of steepest ascent on the loss
surface.
● To minimize loss, move in the opposite direction of the gradient.
● Negative gradient points in the direction of steepest descent on the loss surface.
representations for the purpose of finding patterns.
Perceptron
● A perceptron is an artificial neuron
● It is the simplest possible Neural Network
● The initial perceptron design involved processing multiple binary inputs by multiplying
each input with a corresponding weight, summing the results along with a bias
parameter to yield a binary output (0 or 1).
● Perceptrons are linear functions:
○ Too simple and abstract to link the outputs of one perceptron to the inputs of
the next in a large network.
○ Composing together linear functions will only ever give you another linear
function. We’re not creating models that can learn non-linear functions.
Activation function
● An activation function is a scalar function (a function from a number to another
number) which we apply to the output of a perceptron after all the weighted inputs
have been combined.
● Typically, the activation function is nonlinear, which allows the network to learn
complex patterns in data.
● Not using an activation function is also called using a linear activation (linear
regression model).
Activation function Plot Goal
Linear activation Also known as the "no activation" function"
function (multiplied x1.0), where the activation is
proportional to the input. The function doesn't
do anything to the weighted sum of the input, it
simply spits out the value it was given.
, Binary Depends on a threshold value that decides
Non-linear step whether a neuron should be activated or not.
Sigmoid Converts input into a probability between 0 and
1; often used in binary classification.
Softmax Converts input into categorical probability
distribution by compressing multiple inputs into
a range between 0 and 1; used in multi-class
classification.
Tanh Similar to sigmoid but ranges from -1 to 1;
useful in hidden layers of a neural network.
ReLU Sets every negative input to zero, and keeps
everything else the same. Allows only positive
values to pass through; helps with non-linear
complex mappings and vanishing gradient
problems.
Leaky Similar to ReLU but allows a small, non-zero
ReLU gradient when the unit is not active; helps
prevent dead neurons.
Neural network
● Any arrangement of perceptrons and nonlinearities.
,Multilayer perceptron (MLP)
● A fully connected feed-forward artificial neural network with at least three layers
(input, output, and at least one hidden layer)
● Consists of:
○ Layer of hidden units in the middle, each of which acts as a perceptron
with a non-linearity (activation function), connecting to all input nodes.
○ One or more output nodes, connecting to all nodes in the hidden layer.
● Features:
1. There are no cycles, the network “feeds forward” from input to output.
2. Nodes in the same layer are not connected to each other, or to any other
layer than the previous one.
3. Each layer is fully connected to the previous layer, every node in one layer
connects to every node in the layer before it.
● Note: Every orange and blue line in this picture represents one parameter of
the model.
● Although perceptrons can only understand linear relationships between the input and
output data provided, MLPs are a neural network where the mapping between inputs
and output is non-linear.
● It learns weights in the process of backpropagation.
Finding good weights
Loss function
● Begin by determining a loss function to assess model performance.
● Seek weights (model parameters) that result in minimal loss over the data.
● Lower loss corresponds to a better-performing model.
● Note: It’s nice if the loss is zero when the prediction is perfect, but this isn’t required.
● Can be defined for a single instance or for all instances in the data. Usually, the loss
over the whole data is just the average loss over all instances.
machine learning loss function
algorithm
regression squared distance between y normal distribution
errors and t (fixed variance)
absolute take the absolute laplace distribution
errors magnitude of the (fixed variance)
difference between
, network output and the
target value
classification log loss binary cross-entropy Bernoulli distribution
log loss cross-entropy Categorical
distribution
hinge loss <none>
Loss Landscape
● Model space consists of all possible models, forming a plane.
● Loss can be visualized as a surface above this plane.
● Every point on the plane corresponds to a specific set of
weights (θ), and the loss function defines the loss for each
point.
Optimization Goal
● Objective is to navigate the loss surface and find a set of
weights (θ) that minimizes the loss on the chosen dataset:
● Use calculus to identify the lowest point on the loss surface.
● In one dimension, derivatives indicate the slope of a tangent
line, representing how much the function rises.
● For multiple dimensions, compute partial derivatives (derivatives with respect to
one of those variables, with the others held constant), forming the gradient.
● The gradient is a vector pointing in the direction of steepest ascent on the loss
surface.
● To minimize loss, move in the opposite direction of the gradient.
● Negative gradient points in the direction of steepest descent on the loss surface.
