Lecture 1
Training set – for training the algorithm:
Training set: train the algorithm
Validation set: optimize hyperparameters
Test set: evaluate performance on the test set
Introduction:
What is deep learning:
AI:
-
Hard-code knowledge about the world in formal languages.
-
People struggle to devise formal rules with enough complexity to accurately describe
world knowledge.
Machine learning:
- Acquire their own knowledge by extracting patterns from raw data.
- Performance of simple ML algorithms depends heavily on the representation of the
data (features).
- AI is larger than machine learning because you might also hard code, but in machine
learning, the machine itself learns this kind of information.
- Feature extraction is a crucial step. What might lead to the price of a house. Square
meter, number of rooms, garden, etc. These all give an idea about the price of the
house. These are the features. Feature extraction is very important because it has
impact on how good the target variable is predicted. If we fail to extract reasonable
features, we fail to predict accurately.
,Representation learning:
- Use ML not only to discover mappings from representation to output, but to learn
the representation itself.
Deep learning:
- Solves problem representation learning: introduces representations that are
expressed in terms of other, simpler representations.
- Enables computer to build complex concepts out of simple ones.
- Depth enables the computer to learn a multistep computer program.
History of deep learning: apparently ‘new’ technology with a ‘history’:
The 3 waves of development:
- 1940 – 1960: Cybernetics
o McCulloch-Pitts Neuron: early model of brain function. A neuron itself does
not learn, but when they come together, they can do a lot.
o Perceptron: digital version of a neuron.
o Adaptive linear element (ADLINE): predicting continuous variables
- 1980 – 1990: Connectionism
o Distributed representation
o Backpropagation: essence of learning in the computers. This is where the
algorithm updates its parameters.
- 2006 - …: Deep learning: it has many layers
Artificial neural networks (ANNs):
- Engineered systems inspired by the brain
- One of the names DL has gone by
Motivation:
1. The proof by example that intelligent behavior is possible –> reverse engineer the
computational principles behind it and duplicate its functionality.
2. ML models that help us understand the principles that underlie human intelligence
→ shed light on basic scientific questions.
Deep learning (DL) vs ANNs:
- Appeals to more general principle of learning multiple levels of composition (i.e.,
multiple layers that create ‘depth’).
,Layered structure – primate brain:
The things we see with our eyes goes to the v1 which is the edges of the first hidden layer of
the DL network. Then it goes through more layers and eventually it categorizes. PFC is the
region where we do the cognitive decision. When we have the decision, it is sent to the
motor cortex and then that sends it to the muscles. So, the deep learning layers is inspired
by this structure. Each layer becomes more complex and in the end a decision is made.
We expect a complex network, with millions of parameters to be optimized. It first extracts
complex features and then it makes a decision. It needs to optimize the features. So, it
needs big data.
, CPU vs GPU:
DL frameworks (libraries that you can use):
The perceptron:
What is a perceptron:
- Most basic single-layer NN → typically used for binary classification problems (1 or 0,
yes or no)
- Data needs to be linearly separable (if the decision boundary is non-linear, the
perceptron can’t be used)
- Goal: find a line that splits the data/observations
Bias term: describes the threshold. If the weighted sum of the input is higher or equal to the
threshold, it fires, it is 1. The line that you fit should always go through the original (0,0)
location when not using a bias term). If you do use a bias term, you can fit your line at
different line intersections.
Activation function (step activation function):
Training set – for training the algorithm:
Training set: train the algorithm
Validation set: optimize hyperparameters
Test set: evaluate performance on the test set
Introduction:
What is deep learning:
AI:
-
Hard-code knowledge about the world in formal languages.
-
People struggle to devise formal rules with enough complexity to accurately describe
world knowledge.
Machine learning:
- Acquire their own knowledge by extracting patterns from raw data.
- Performance of simple ML algorithms depends heavily on the representation of the
data (features).
- AI is larger than machine learning because you might also hard code, but in machine
learning, the machine itself learns this kind of information.
- Feature extraction is a crucial step. What might lead to the price of a house. Square
meter, number of rooms, garden, etc. These all give an idea about the price of the
house. These are the features. Feature extraction is very important because it has
impact on how good the target variable is predicted. If we fail to extract reasonable
features, we fail to predict accurately.
,Representation learning:
- Use ML not only to discover mappings from representation to output, but to learn
the representation itself.
Deep learning:
- Solves problem representation learning: introduces representations that are
expressed in terms of other, simpler representations.
- Enables computer to build complex concepts out of simple ones.
- Depth enables the computer to learn a multistep computer program.
History of deep learning: apparently ‘new’ technology with a ‘history’:
The 3 waves of development:
- 1940 – 1960: Cybernetics
o McCulloch-Pitts Neuron: early model of brain function. A neuron itself does
not learn, but when they come together, they can do a lot.
o Perceptron: digital version of a neuron.
o Adaptive linear element (ADLINE): predicting continuous variables
- 1980 – 1990: Connectionism
o Distributed representation
o Backpropagation: essence of learning in the computers. This is where the
algorithm updates its parameters.
- 2006 - …: Deep learning: it has many layers
Artificial neural networks (ANNs):
- Engineered systems inspired by the brain
- One of the names DL has gone by
Motivation:
1. The proof by example that intelligent behavior is possible –> reverse engineer the
computational principles behind it and duplicate its functionality.
2. ML models that help us understand the principles that underlie human intelligence
→ shed light on basic scientific questions.
Deep learning (DL) vs ANNs:
- Appeals to more general principle of learning multiple levels of composition (i.e.,
multiple layers that create ‘depth’).
,Layered structure – primate brain:
The things we see with our eyes goes to the v1 which is the edges of the first hidden layer of
the DL network. Then it goes through more layers and eventually it categorizes. PFC is the
region where we do the cognitive decision. When we have the decision, it is sent to the
motor cortex and then that sends it to the muscles. So, the deep learning layers is inspired
by this structure. Each layer becomes more complex and in the end a decision is made.
We expect a complex network, with millions of parameters to be optimized. It first extracts
complex features and then it makes a decision. It needs to optimize the features. So, it
needs big data.
, CPU vs GPU:
DL frameworks (libraries that you can use):
The perceptron:
What is a perceptron:
- Most basic single-layer NN → typically used for binary classification problems (1 or 0,
yes or no)
- Data needs to be linearly separable (if the decision boundary is non-linear, the
perceptron can’t be used)
- Goal: find a line that splits the data/observations
Bias term: describes the threshold. If the weighted sum of the input is higher or equal to the
threshold, it fires, it is 1. The line that you fit should always go through the original (0,0)
location when not using a bias term). If you do use a bias term, you can fit your line at
different line intersections.
Activation function (step activation function):
