Artificial Intelligence

Artificial Intelligence
Artificial Neural
Network
Pham Viet Cuong
Dept. Control Engineering & Automation, FEEE
Ho Chi Minh City University of Technology
Artificial Neural Networks
✓ Computing systems inspired by biological neural networks
✓ Consists of several processing elements that receive inputs and deliver
outputs
✓ "Learn" to perform tasks by considering examples
✓ Be able to model nonlinear processes
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 2
✓ Applications:
❖ Natural Language Processing
▪ Text classification
▪ Language generation
▪ Document summarization
▪ Machine translation
▪ Speech recognition
▪ Character recognition
▪ Spell checking
✓ Applications:
❖ System identification and control
▪ Vehicle control
▪ Process control
▪ Auto-piloting
▪ Autonomous driving cars
▪ Robot navigation
▪ Component fault detection and simulation
▪ Chip failure analysis
▪ Trajectory prediction
▪ Petroleum exploration
✓ Applications:
❖ Time series forecasting and prediction
▪ Weather forecasting
▪ Stock market prediction
▪ Cost models
❖ Medical
▪ Cancers detection
▪ Medical image analysis
▪ EEGs
▪ Drug design
✓ Applications:
❖ Image/video/audio analysis
▪ Image recognition
▪ Image compression
▪ Radar and sonar image classification
▪ Signature verification
✓ Applications:
❖ Education:
▪ Adaptive learning software
▪ Student performance modeling
▪ Personality profiling
❖ Bussiness analytics
▪ Customer behavior modeling
▪ Customer segmentation
▪ Market research
❖ Banking
▪ Credit and loan application evaluation
▪ Fraud and risk evaluation
✓ Biological neural network
✓ Artificial neural network
w1
w2
wn
out = f(w1in1 + w2in2 + . . . + wninn + bias)
✓ Activation function
sigmoid
✓ Example: self-driving car
❖ Input (raw data/features)
❖ Output (steering angle, throttle, brake)
❖ Hidden layers
✓ Why nonlinear activation function?
✓ Why bias?
✓ Useful links
❖ Link 1
❖ Link 2
❖ Link 3
✓ Error back propagation Forward
❖ Framework
❖ Loss function:
Backward
➢ Squared L2 norm
➢ Angle between 2 vectors
➢ Cross entropy
✓ Error back propagation
❖ Loss function:
➢ Euclidean norm (2-norm)
➢ Angle between 2 vectors

➢ Cross entropy
❖ Loss function:
➢ Euclidean norm (2-norm) Cross entropy
❖ Optimization problem
❖ Gradient descent
𝑑𝑓
𝑥←𝑥−
𝑑𝑥
𝑑𝑓
𝑥 ←𝑥−𝜇
𝑑𝑥
𝜇: learning rate
𝑥2
𝜕𝐿
𝑤 ←𝑤−𝜇 𝑥1
𝜕𝑤
❖ Example Forward
𝜕𝐿
𝑤 ←𝑤−𝜇
𝜕𝑤
Backward
❖ Example
𝑛𝑒𝑡ℎ1 = 𝑤1 𝑖1 + 𝑤2 𝑖2 + 𝑏1
𝑜𝑢𝑡ℎ1 = 𝑓 𝑛𝑒𝑡ℎ1
1
𝑓 𝑥 =
1 + 𝑒 −𝑥
1 2 2
𝐸= 𝑡𝑎𝑟𝑔𝑒𝑡𝑜1 − 𝑜𝑢𝑡𝑜1 + 𝑡𝑎𝑟𝑔𝑒𝑡𝑜2 − 𝑜𝑢𝑡𝑜2
2
𝜕𝐿
𝑤 ←𝑤−𝜇
𝜕𝑤
𝜕𝐸
𝜕𝑤5
❖ Chain rule
𝜕𝐸 𝜕𝐸 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1
=
𝜕𝑤5 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝑤5
❖ Example
=
𝑛𝑒𝑡𝑜1 = 𝑤5 𝑜𝑢𝑡ℎ1 + 𝑤6 𝑜𝑢𝑡ℎ2 + 𝑏2
𝑜𝑢𝑡𝑜1 = 𝑓 𝑛𝑒𝑡𝑜1
1
𝑓 𝑥 =
1 + 𝑒 −𝑥
1 2 2
𝐸= 𝑡𝑎𝑟𝑔𝑒𝑡𝑜1 − 𝑜𝑢𝑡𝑜1 + 𝑡𝑎𝑟𝑔𝑒𝑡𝑜2 − 𝑜𝑢𝑡𝑜2
2
❖ Chain rule
=
w5 netO1 outO1 E
❖ Chain rule
=
𝜕𝐸 𝜕𝐸 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑛𝑒𝑡ℎ1

=
𝜕𝑤1 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑛𝑒𝑡ℎ1 𝜕𝑤1
❖ Chain rule
𝜕𝐸 𝜕𝐸 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑛𝑒𝑡ℎ1
=
𝜕𝑤1 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑛𝑒𝑡ℎ1 𝜕𝑤1
w1 neth1 outh1 netO1 outO1 E
netO2 outO2
𝜕𝐸 𝜕𝐸 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝐸 𝜕𝑜𝑢𝑡𝑜2 𝜕𝑛𝑒𝑡𝑜2 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑛𝑒𝑡ℎ1

= +
𝜕𝑤1 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑜𝑢𝑡𝑜2 𝜕𝑛𝑒𝑡𝑜2 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑛𝑒𝑡ℎ1 𝜕𝑤1
❖ Homework 2: Find derivative formulas for a NN
▪ 2 inputs
▪ 2 hidden layers, each layer has 3 neurals
▪ 3 outputs
w7 w16
w1
❖ Vanishing gradient problem 𝜕𝐸
𝜕𝑤1
=
𝜕𝐸 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1
𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝑜𝑢𝑡ℎ1
+
𝜕𝐸 𝜕𝑜𝑢𝑡𝑜2 𝜕𝑛𝑒𝑡𝑜2 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑛𝑒𝑡ℎ1
𝜕𝑜𝑢𝑡𝑜2 𝜕𝑛𝑒𝑡𝑜2 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑛𝑒𝑡ℎ1 𝜕𝑤1
𝜕𝐿
𝑤 ←𝑤−𝜇
𝜕𝑤
✓ Overfitting: high variance
✓ Underfitting: high bias
✓ Overfitting vs. Underfitting
✓ Reasons
❖ Underfitting (simplifying assumption)
▪ Too simple model
▪ Not enough training
❖ Overfitting
▪ Too complicated model
▪ Not enough training data
✓ Countermeasures for Underfitting
❖ Better model
❖ More training
✓ Countermeasures for Overfitting
❖ Simplifying model
❖ Removing features from data
❖ More data
▪ ImageNet Large Scale Visual Recognition Challenge
▪ 1000 classes
▪ 1.2 M images
❖ Data augmentation
▪ Extracting patches
▪ Reflection
▪ Reflection
▪ Rotation
▪ Reflection
▪ Rotation
▪ Altering intensities of RGB channels
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT
?
38
▪ Reflection
▪ Rotation
▪ Altering intensities of RGB channels
▪ and more
▪ AlexNet: 2048 times
✓ Countermeasures
❖ Early stopping
▪ Training set
▪ Test set
▪ Validation set
✓ Reasons
✓ Countermeasures
❖ Early stopping
▪ Training set
▪ Test set
▪ Validation set
✓ Countermeasures
❖ Drop out
▪ Co-adaptation
✓ Gradient descent
❖ 2D illustration
❖ Stochastic gradient descent Forward
❖ Batch gradient descent
L
𝜕𝐿
𝑤 ←𝑤−𝜇
𝜕𝑤
Backward
❖ Stochastic gradient descent
▪ Update: each training example
▪ Online
▪ Unstable
▪ Can avoid local minima
▪ Update: all training dataset
▪ More stable
▪ May prematurely converge
❖ 2D illustration
❖ Stochastic gradient descent
❖ Mini-batch gradient descent
▪ Update: subset of training set
▪ Mini-batch size: 16 - 256
▪ Best of both worlds
❖ Data point/example/sample/instance/input vector/feature
vector/observation
❖ Batch size
❖ Epoch: 10, 100, 500, 1000, and larger
✓ References
❖ https://en.wikipedia.org/wiki/Neural_circuit
❖ https://en.wikipedia.org/wiki/Artificial_neural_network
❖ https://www.securityinfowatch.com/video-surveillance/video-
analytics/article/21069937/deep-learning-to-the-rescue
❖ and others

Artificial Intelligence

Uploaded by

Copyright:

Available Formats

Artificial Intelligence

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Artificial Intelligence

Uploaded by

Copyright:

Available Formats

Artificial Intelligence

➢ Angle between 2 vectors

𝑛𝑒𝑡𝑜1 = 𝑤5 𝑜𝑢𝑡ℎ1 + 𝑤6 𝑜𝑢𝑡ℎ2 + 𝑏2

𝜕𝐸 𝜕𝐸 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑛𝑒𝑡ℎ1

w1 neth1 outh1 netO1 outO1 E

𝜕𝐸 𝜕𝐸 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝐸 𝜕𝑜𝑢𝑡𝑜2 𝜕𝑛𝑒𝑡𝑜2 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑛𝑒𝑡ℎ1

You might also like