Summary Deep Learning
Table of Contents
1. Neural Network Fundamentals ................................................................................................................................ 2
1.1. Introduction ............................................................................................................................................................. 2
1.2. A brief history .......................................................................................................................................................... 3
1.3. Software .................................................................................................................................................................. 3
1.4. The perceptron ......................................................................................................................................................... 4
1.5. Multilayer perceptron (MLP) networks ..................................................................................................................... 5
1.6. Delving deeper into neural networks ........................................................................................................................ 6
1.7. Double descent: revisiting bias-variance ................................................................................................................... 9
2. Convolutional neural networks .............................................................................................................................. 11
2.1. The convolutional architecture ............................................................................................................................... 11
2.2. Preventing overfitting ............................................................................................................................................ 13
2.3. Delving deeper into convolutional neural networks................................................................................................ 14
2.4. Transfer learning .................................................................................................................................................... 16
2.5. Some early “generative” applications of pretrained CNNs ...................................................................................... 16
2.6. Opening the black box ............................................................................................................................................ 17
3. Representational learning ..................................................................................................................................... 18
3.1. Embeddings for text ............................................................................................................................................... 18
3.2. Generalising embeddings ....................................................................................................................................... 19
3.3. Auto-encoders ........................................................................................................................................................ 20
4. Recurrent Neural Networks ................................................................................................................................... 22
4.1. Sequential data ...................................................................................................................................................... 22
4.2. Window based models ........................................................................................................................................... 23
4.3. The recurrent architecture...................................................................................................................................... 24
4.4. Delving deeper into the recurrent architecture ....................................................................................................... 25
5. Generative AI ........................................................................................................................................................ 27
5.1. Generative adversarial networks ............................................................................................................................ 27
5.2. Diffusion models .................................................................................................................................................... 28
5.3. Image models and tools.......................................................................................................................................... 31
5.4. Transformers .......................................................................................................................................................... 31
5.5. Language models and tools .................................................................................................................................... 37
6. Reinforcement learning ......................................................................................................................................... 38
6.1. Introduction ........................................................................................................................................................... 38
6.2. Value learning ........................................................................................................................................................ 40
6.3. Deep Q networks.................................................................................................................................................... 41
6.4. Policy learning ........................................................................................................................................................ 42
6.5. Multi-agent environments ...................................................................................................................................... 42
6.6. Reinforcement Learning and Large Language Models ............................................................................................. 43
6.7. Software ................................................................................................................................................................ 43
7. Graph neural networks .......................................................................................................................................... 44
7.1. Introduction ........................................................................................................................................................... 44
7.2. Basic graph mining ................................................................................................................................................. 44
7.3. Learning representations from graphs .................................................................................................................... 47
7.4. Graph Neural Networks (GNNs) .............................................................................................................................. 47
7.5. Examples and practice ............................................................................................................................................ 49
,1. Neural Network Fundamentals
1.1. Introduction
1) From if-then rules to deep learning
• Expert-based rules: are obtained from business experts
- Subjective, but not necessarily inferior
- Still remains popular in many settings
• Data-based models: are obtained from data using analytical techniques
- Ex.: Rule induction, decision trees
- White vs. black box models:
§ White box = even managers can understand results
§ Black box = more complex, more difficult to understand, but higher performance
- Shallow vs. deep models:
§ Shallow = simple data-based models with low number of parameters & layers
§ Deep = highly complex architectures featuring a lot of parameters & layers
• Structured, “tabular” data is used for shallow modelling
- Usually performs well
- Ex.: logistic regression, SVM’s, decision trees, ensemble models
2) Artificial Neural Networks (ANNs)
• As data complexity increases, ANNs tend to be more performant in capturing patterns
- For very complex input data, ANNs with a small number of hidden layers are inaccurate
- For this type of data, ANNs with more layers are most performant
• ANNs have been around for quite some time (1960s)
à But the revival was triggered by:
- Increase of computing power: the emergence of Graphic Processing Units (GPUs)
- Huge volumes of new data types: images, text, audio, video…
- Software support: TensorFlow, PyTorch, Keras…
- New use cases: medical, finance, logistics, retail…
“The goal is to create algorithms that can take in very unstructured data (images, audio
waves, text) and predict the properties of those inputs” – Andrew Ng
3) Comparison
Traditional algorithms Deep learning
Accuracy Fair to excellent Good to excellent
Training time Short to medium Medium to very long
Data requirements Limited High
Feature engineering Manual Automatic
Hyperparameters Few to some Many
Interpretability High to reasonable Low
Cost & operational efficiency Low to reasonable Reasonable to high
,1.2. A brief history
• (1940s) The beginnings: “an electronic brain”
- Since the beginning of computing, researchers thought about an intelligent machine
- Alan Turing laid out requirements to assess whether a machine could be called
intelligent
• (1950s) A basic building block: “the perceptron”
= neuron-based linear classifier with a learning rule
- Simplified mathematical representation of a neuron
- With a learning procedure that would converge to the correct, desired solution
• (1960-80s): From a golden age to the first AI winter
- Marvin Minsky less convinced of perceptron: could not learn simple XOR
- This halted research on neural nets
• (1980s): The return of artificial neural networks
- Shows that neural networks with many hidden layers could be trained by a procedure
called backpropagation
- Backpropagation works by starting from a network’s error and “back-propagating” this
throughout all its layers to adjust their parameters
- First image results with Convolutional Neural Networks (CNNs)
• (1990s-2000s): A second winter arrives
- MLP approach didn’t scale well to larger problems (lack of computing power)
- Meanwhile Support Vector Machine (SVM) was rapidly taking the centre-stage as the
method of choice
• (Early 2000s): Deep learning is born
- Introduction of “unsupervised pretraining” and “deep belief nets”
- Large database (ImageNet) was created
- BUT: still error rates of 25% L
• (2010s): Deep learning unleashed
- Error rates are halved
- Many innovations followed this result
§ Improved architectures (CNNs, RCNNs, GANs, RNNs…)
§ Software support (TensorFlow, PyTorch, Keras…)
§ New regularisation techniques (dropout, batch normalization)
§ New optimisers (Stochastic Gradient Descent (SGD), RMSprop, ADAM…)
• (2020s): What’s ahead
- Many are convinced that we are amid a great AI awakening
- The field is moving at rapid pace, still exploring the full extent of deep learning
- Especially transformer & attention-based models (GPT) are expanding
- BUT: important to not get carried away (research still ongoing)
- Some predict a 3rd AI winter is coming
à BUT: relatively unlikely given current data & computing power
1.3. Software
• Software support is one of the key enablers of deep learning
• Torch è PyTorch:
- Torch is a computational machine learning framework
- PyTorch, a python version of Torch, was open-sourced by Facebook (2017)
à Offers dynamic computational graphs & easily builds complex architectures
, • Theano è TensorFlow
- Theano: one of the oldest deep-learning frameworks
à Focus on fast handling of multidimensional arrays
- TensorFlow: created by google to replace Theano
• Keras
- Keras1 is a deep learning library
à Provides high-level, easy API inspired by Torch
- Keras3 went back to multi-framework
1.4. The perceptron
• The perceptron is based on how we think the human brain works
à A collection of artificial “neurons” is created and connected, allowing them to send
messages to each other
1) Components of the perceptron
• Input neurons (𝑥! , 𝑥" , 𝑥# ): data fed into the model
- Data should be normalised
• Weights (𝑤! , 𝑤" , 𝑤# ): values determining the importance of each input
- Determining weights:
§ Simple statistical models: closed-form analytical formulas to determine the
optimum parameter estimates exist
à Ex.: linear regression
§ Nonlinear models: the parameter estimates must be determined numerically,
using an iterative algorithm
à Ex.: neural networks
- Weights remain fixed once they are trained
- Activation depends on the instance given
• Bias weight (𝑤$ ): additional input unit with output fixed at 1 and its own weight
- Comparable to “intercept” in a statistical model
• Summation (∑): linear combination of weighted inputs & bias together
• Activation function (𝑓(. )): determines final output based on the total sum
• Output (arrow): the final result produced by the perceptron
- Rosenblatt’s original approach: 𝑂(𝑥) = 1 𝑖𝑓 ∑ 𝑤% 𝑥% + 𝑤$ > 0 𝑒𝑙𝑠𝑒 − 1
à Not easy to apply gradient descent on thresholded output function
- Logistic regression: 𝑂(𝑥) = 𝑠𝑖𝑔𝑚𝑜𝑖𝑑(∑ 𝑤% 𝑥% + 𝑤$ ) = 19
1 + 𝑒 &(∑ )! *! +)" )
2) Gradient descent
• Gradient descent = process of minimising a function by following gradients of the loss
function (𝐿)
- Involves 1st partial derivate of the loss function at a given point
/0
- 𝑤%,.+! = 𝑤%,. − 𝜆 /)
!
Table of Contents
1. Neural Network Fundamentals ................................................................................................................................ 2
1.1. Introduction ............................................................................................................................................................. 2
1.2. A brief history .......................................................................................................................................................... 3
1.3. Software .................................................................................................................................................................. 3
1.4. The perceptron ......................................................................................................................................................... 4
1.5. Multilayer perceptron (MLP) networks ..................................................................................................................... 5
1.6. Delving deeper into neural networks ........................................................................................................................ 6
1.7. Double descent: revisiting bias-variance ................................................................................................................... 9
2. Convolutional neural networks .............................................................................................................................. 11
2.1. The convolutional architecture ............................................................................................................................... 11
2.2. Preventing overfitting ............................................................................................................................................ 13
2.3. Delving deeper into convolutional neural networks................................................................................................ 14
2.4. Transfer learning .................................................................................................................................................... 16
2.5. Some early “generative” applications of pretrained CNNs ...................................................................................... 16
2.6. Opening the black box ............................................................................................................................................ 17
3. Representational learning ..................................................................................................................................... 18
3.1. Embeddings for text ............................................................................................................................................... 18
3.2. Generalising embeddings ....................................................................................................................................... 19
3.3. Auto-encoders ........................................................................................................................................................ 20
4. Recurrent Neural Networks ................................................................................................................................... 22
4.1. Sequential data ...................................................................................................................................................... 22
4.2. Window based models ........................................................................................................................................... 23
4.3. The recurrent architecture...................................................................................................................................... 24
4.4. Delving deeper into the recurrent architecture ....................................................................................................... 25
5. Generative AI ........................................................................................................................................................ 27
5.1. Generative adversarial networks ............................................................................................................................ 27
5.2. Diffusion models .................................................................................................................................................... 28
5.3. Image models and tools.......................................................................................................................................... 31
5.4. Transformers .......................................................................................................................................................... 31
5.5. Language models and tools .................................................................................................................................... 37
6. Reinforcement learning ......................................................................................................................................... 38
6.1. Introduction ........................................................................................................................................................... 38
6.2. Value learning ........................................................................................................................................................ 40
6.3. Deep Q networks.................................................................................................................................................... 41
6.4. Policy learning ........................................................................................................................................................ 42
6.5. Multi-agent environments ...................................................................................................................................... 42
6.6. Reinforcement Learning and Large Language Models ............................................................................................. 43
6.7. Software ................................................................................................................................................................ 43
7. Graph neural networks .......................................................................................................................................... 44
7.1. Introduction ........................................................................................................................................................... 44
7.2. Basic graph mining ................................................................................................................................................. 44
7.3. Learning representations from graphs .................................................................................................................... 47
7.4. Graph Neural Networks (GNNs) .............................................................................................................................. 47
7.5. Examples and practice ............................................................................................................................................ 49
,1. Neural Network Fundamentals
1.1. Introduction
1) From if-then rules to deep learning
• Expert-based rules: are obtained from business experts
- Subjective, but not necessarily inferior
- Still remains popular in many settings
• Data-based models: are obtained from data using analytical techniques
- Ex.: Rule induction, decision trees
- White vs. black box models:
§ White box = even managers can understand results
§ Black box = more complex, more difficult to understand, but higher performance
- Shallow vs. deep models:
§ Shallow = simple data-based models with low number of parameters & layers
§ Deep = highly complex architectures featuring a lot of parameters & layers
• Structured, “tabular” data is used for shallow modelling
- Usually performs well
- Ex.: logistic regression, SVM’s, decision trees, ensemble models
2) Artificial Neural Networks (ANNs)
• As data complexity increases, ANNs tend to be more performant in capturing patterns
- For very complex input data, ANNs with a small number of hidden layers are inaccurate
- For this type of data, ANNs with more layers are most performant
• ANNs have been around for quite some time (1960s)
à But the revival was triggered by:
- Increase of computing power: the emergence of Graphic Processing Units (GPUs)
- Huge volumes of new data types: images, text, audio, video…
- Software support: TensorFlow, PyTorch, Keras…
- New use cases: medical, finance, logistics, retail…
“The goal is to create algorithms that can take in very unstructured data (images, audio
waves, text) and predict the properties of those inputs” – Andrew Ng
3) Comparison
Traditional algorithms Deep learning
Accuracy Fair to excellent Good to excellent
Training time Short to medium Medium to very long
Data requirements Limited High
Feature engineering Manual Automatic
Hyperparameters Few to some Many
Interpretability High to reasonable Low
Cost & operational efficiency Low to reasonable Reasonable to high
,1.2. A brief history
• (1940s) The beginnings: “an electronic brain”
- Since the beginning of computing, researchers thought about an intelligent machine
- Alan Turing laid out requirements to assess whether a machine could be called
intelligent
• (1950s) A basic building block: “the perceptron”
= neuron-based linear classifier with a learning rule
- Simplified mathematical representation of a neuron
- With a learning procedure that would converge to the correct, desired solution
• (1960-80s): From a golden age to the first AI winter
- Marvin Minsky less convinced of perceptron: could not learn simple XOR
- This halted research on neural nets
• (1980s): The return of artificial neural networks
- Shows that neural networks with many hidden layers could be trained by a procedure
called backpropagation
- Backpropagation works by starting from a network’s error and “back-propagating” this
throughout all its layers to adjust their parameters
- First image results with Convolutional Neural Networks (CNNs)
• (1990s-2000s): A second winter arrives
- MLP approach didn’t scale well to larger problems (lack of computing power)
- Meanwhile Support Vector Machine (SVM) was rapidly taking the centre-stage as the
method of choice
• (Early 2000s): Deep learning is born
- Introduction of “unsupervised pretraining” and “deep belief nets”
- Large database (ImageNet) was created
- BUT: still error rates of 25% L
• (2010s): Deep learning unleashed
- Error rates are halved
- Many innovations followed this result
§ Improved architectures (CNNs, RCNNs, GANs, RNNs…)
§ Software support (TensorFlow, PyTorch, Keras…)
§ New regularisation techniques (dropout, batch normalization)
§ New optimisers (Stochastic Gradient Descent (SGD), RMSprop, ADAM…)
• (2020s): What’s ahead
- Many are convinced that we are amid a great AI awakening
- The field is moving at rapid pace, still exploring the full extent of deep learning
- Especially transformer & attention-based models (GPT) are expanding
- BUT: important to not get carried away (research still ongoing)
- Some predict a 3rd AI winter is coming
à BUT: relatively unlikely given current data & computing power
1.3. Software
• Software support is one of the key enablers of deep learning
• Torch è PyTorch:
- Torch is a computational machine learning framework
- PyTorch, a python version of Torch, was open-sourced by Facebook (2017)
à Offers dynamic computational graphs & easily builds complex architectures
, • Theano è TensorFlow
- Theano: one of the oldest deep-learning frameworks
à Focus on fast handling of multidimensional arrays
- TensorFlow: created by google to replace Theano
• Keras
- Keras1 is a deep learning library
à Provides high-level, easy API inspired by Torch
- Keras3 went back to multi-framework
1.4. The perceptron
• The perceptron is based on how we think the human brain works
à A collection of artificial “neurons” is created and connected, allowing them to send
messages to each other
1) Components of the perceptron
• Input neurons (𝑥! , 𝑥" , 𝑥# ): data fed into the model
- Data should be normalised
• Weights (𝑤! , 𝑤" , 𝑤# ): values determining the importance of each input
- Determining weights:
§ Simple statistical models: closed-form analytical formulas to determine the
optimum parameter estimates exist
à Ex.: linear regression
§ Nonlinear models: the parameter estimates must be determined numerically,
using an iterative algorithm
à Ex.: neural networks
- Weights remain fixed once they are trained
- Activation depends on the instance given
• Bias weight (𝑤$ ): additional input unit with output fixed at 1 and its own weight
- Comparable to “intercept” in a statistical model
• Summation (∑): linear combination of weighted inputs & bias together
• Activation function (𝑓(. )): determines final output based on the total sum
• Output (arrow): the final result produced by the perceptron
- Rosenblatt’s original approach: 𝑂(𝑥) = 1 𝑖𝑓 ∑ 𝑤% 𝑥% + 𝑤$ > 0 𝑒𝑙𝑠𝑒 − 1
à Not easy to apply gradient descent on thresholded output function
- Logistic regression: 𝑂(𝑥) = 𝑠𝑖𝑔𝑚𝑜𝑖𝑑(∑ 𝑤% 𝑥% + 𝑤$ ) = 19
1 + 𝑒 &(∑ )! *! +)" )
2) Gradient descent
• Gradient descent = process of minimising a function by following gradients of the loss
function (𝐿)
- Involves 1st partial derivate of the loss function at a given point
/0
- 𝑤%,.+! = 𝑤%,. − 𝜆 /)
!
