Introduction to Deep Learning (800883-B-6)
Summary Lectures Final Exam
CSAI year 3
Written by Saskia Kriege
,Lecture 1 – Introduction and MultiLayer Perceptron
Neural Networks
Train = changing parameters
Trying to optimize the black box by changing numbers (parameters), done by understanding
the error that the model makes
Use error to change parameters of the network, to estimate the actual function of the network
Universal Function Approximator
Approximate functions
Y = f(x) → output y, input y, figure out the function f(x)
Input x → NN (approximate an unknown function y = f(x)) → output y
History and Context – NOT EXAM MATERIAL
Ramon y Cajal → connectionist approach of how the brain works
Individual tissues doing individual ones doing computations by themselves → neurons
Those neurons were connected, connections changed how they were firing
Emerging from this is intelligence
McCulloch and Pitts
Computers were emerging, idea to build mathematical models of this idea of the brain
Logic Gates based on connectionist approach, little units putted together gives a more
complex thing
Based on Logic, input and output only 1’s and 0’s.
Rosenblatt
The Perceptron → idea we are still using for NN
Changed → how they’re trained and put together
Perceptron learning parameters (weights)
Weight + input gives a certain output
Perceptrons and the AI Winter
Minsky and Papert
You cannot solve simple problems with this perceptron → not taking us closer to what the
brain does
Basic problems cannot be solved (sort problem)
People stopped believing in AI, funding disappeared from research
The AI Winter
Some problems could not be solved, using a perceptron was not complete enough.
1980s Boom
Found out how to train network in different ways, got more interesting results
Journals, conferences appeared
Neocognitron – Fukushima → image processing for NN
Backpropagation
,What we use to train networks
Lecun → digit recognition
Another AI Winter → we didn’t have the data and computers to apply the methods we found
out
Big Data
Computers put together with a lot of data
2012 → The cat experiment → neurons learnt to respond to specific stimuli like cats
ImageNet = image database organized according to the WordNet hierarchy, in which each
node of the hierarchy is depicted by hundreds and thousands of images
→ has a lot of biases
AlexNet
Deep CNN trained on ImageNet using GPUs.
Hinton et al.
Generative Adversarial Networks
You have to generate data instead of other way around
Relevant in +- last 5 years
Deep Reinforcement Learning
Neural network
Inputs and expected output changes
How to produce behavior that’s relevant for a particular task
Feed in a frame, calculate error
Deep Learning
Inside AI we have ML (learning from data)
More narrow in ML we have Representation Learning (take the data, model has to figure out
what to do). Transform data into something else
More narrow DL → many layers, each layer trying to extract more abstract features
Practical Deep Learning
CPU vs GPU
GPUs allow for parallelism
CPU → sequential
CPU → each core can do 1 thing at a
time
GPU → you can do as much cores there
are at the same time
DL = multiplications and additions
, GPU allows us to do it more fast, CPU cores are more powerful, but GPU allows to do the
calculations at the same time
Networks require GPU, else it would be too slow
GPU can process many pieces of data at same time
The main difference between CPU and GPU architecture is that a CPU is designed to handle
a wide-range of tasks quickly (as measured by CPU clock speed), but are limited in the
concurrency of tasks that can be running. A GPU is designed to quickly render high-resolution
images and video concurrently.
The Perceptron
A model of a neuron
Dendrites (input) → body of neuron → axon (output)
Then it connects to other neurons through electrical chemical signals, interacting with other
neurons (synapse & synaptic cleft)
A simpler model of a neuron
Incoming dendrites → soma (S) (once threshold is passed), going to axon triggering the rest
Some will increase/decrease in the soma
Incoming dendrites summed together and passed on
Rosenblatt → Perceptron designed to illustrate some of the fundamental properties of
intelligent systems in general, without becoming too deeply enmeshed in the special, and
frequently unknown, conditions which hold for particular biological organisms
Inputs → black box → outputs
Inputs take signals, in the body there is a function that will accumulate those signals by a
summation
Inputs x1, x2, x3
There are weights in the arrows
Those are summed up in the body
Gives y
Linear Classifier → inputting some values and having some weights.
Linearly combining inputs and deciding if they fit given a pattern
Bigger than a threshold → belongs to a certain class
Summary Lectures Final Exam
CSAI year 3
Written by Saskia Kriege
,Lecture 1 – Introduction and MultiLayer Perceptron
Neural Networks
Train = changing parameters
Trying to optimize the black box by changing numbers (parameters), done by understanding
the error that the model makes
Use error to change parameters of the network, to estimate the actual function of the network
Universal Function Approximator
Approximate functions
Y = f(x) → output y, input y, figure out the function f(x)
Input x → NN (approximate an unknown function y = f(x)) → output y
History and Context – NOT EXAM MATERIAL
Ramon y Cajal → connectionist approach of how the brain works
Individual tissues doing individual ones doing computations by themselves → neurons
Those neurons were connected, connections changed how they were firing
Emerging from this is intelligence
McCulloch and Pitts
Computers were emerging, idea to build mathematical models of this idea of the brain
Logic Gates based on connectionist approach, little units putted together gives a more
complex thing
Based on Logic, input and output only 1’s and 0’s.
Rosenblatt
The Perceptron → idea we are still using for NN
Changed → how they’re trained and put together
Perceptron learning parameters (weights)
Weight + input gives a certain output
Perceptrons and the AI Winter
Minsky and Papert
You cannot solve simple problems with this perceptron → not taking us closer to what the
brain does
Basic problems cannot be solved (sort problem)
People stopped believing in AI, funding disappeared from research
The AI Winter
Some problems could not be solved, using a perceptron was not complete enough.
1980s Boom
Found out how to train network in different ways, got more interesting results
Journals, conferences appeared
Neocognitron – Fukushima → image processing for NN
Backpropagation
,What we use to train networks
Lecun → digit recognition
Another AI Winter → we didn’t have the data and computers to apply the methods we found
out
Big Data
Computers put together with a lot of data
2012 → The cat experiment → neurons learnt to respond to specific stimuli like cats
ImageNet = image database organized according to the WordNet hierarchy, in which each
node of the hierarchy is depicted by hundreds and thousands of images
→ has a lot of biases
AlexNet
Deep CNN trained on ImageNet using GPUs.
Hinton et al.
Generative Adversarial Networks
You have to generate data instead of other way around
Relevant in +- last 5 years
Deep Reinforcement Learning
Neural network
Inputs and expected output changes
How to produce behavior that’s relevant for a particular task
Feed in a frame, calculate error
Deep Learning
Inside AI we have ML (learning from data)
More narrow in ML we have Representation Learning (take the data, model has to figure out
what to do). Transform data into something else
More narrow DL → many layers, each layer trying to extract more abstract features
Practical Deep Learning
CPU vs GPU
GPUs allow for parallelism
CPU → sequential
CPU → each core can do 1 thing at a
time
GPU → you can do as much cores there
are at the same time
DL = multiplications and additions
, GPU allows us to do it more fast, CPU cores are more powerful, but GPU allows to do the
calculations at the same time
Networks require GPU, else it would be too slow
GPU can process many pieces of data at same time
The main difference between CPU and GPU architecture is that a CPU is designed to handle
a wide-range of tasks quickly (as measured by CPU clock speed), but are limited in the
concurrency of tasks that can be running. A GPU is designed to quickly render high-resolution
images and video concurrently.
The Perceptron
A model of a neuron
Dendrites (input) → body of neuron → axon (output)
Then it connects to other neurons through electrical chemical signals, interacting with other
neurons (synapse & synaptic cleft)
A simpler model of a neuron
Incoming dendrites → soma (S) (once threshold is passed), going to axon triggering the rest
Some will increase/decrease in the soma
Incoming dendrites summed together and passed on
Rosenblatt → Perceptron designed to illustrate some of the fundamental properties of
intelligent systems in general, without becoming too deeply enmeshed in the special, and
frequently unknown, conditions which hold for particular biological organisms
Inputs → black box → outputs
Inputs take signals, in the body there is a function that will accumulate those signals by a
summation
Inputs x1, x2, x3
There are weights in the arrows
Those are summed up in the body
Gives y
Linear Classifier → inputting some values and having some weights.
Linearly combining inputs and deciding if they fit given a pattern
Bigger than a threshold → belongs to a certain class
