19PEC08 – Advanced Soft

Computing Techniques
 Course Objectives and Course Outcomes
Course Objectives:
• To provide an introduction to the emerging area of intelligent control and optimization.
• To offer a basic knowledge on expert systems, fuzzy logic systems, artificial neural networks
and optimization techniques.
Course Outcomes:
Upon completion of the course, students will be able
– Elucidate the concept of soft computing techniques and their applications.
– Design neural network controller for various system.
– Apply Fuzzy Logic for real world problems.
– Apply Evolutionary Algorithm for obtaining optimal value for real world problem.

– Apply Genetic Algorithm and particle swarm optimization for power electronic optimization
problems.
 19PEC08 – Advanced Soft Computing
Techniques

• UNIT I - INTRODUCTION TO SOFT COMPUTING, ARTIFICIAL

NEURAL NETWORKS

• UNIT II - RECURRENT NEURAL NETWORKS

• UNIT III - FUZZY LOGIC SYSTEM

• UNIT IV - EVOLUTIONARY ALGORITHMS

• UNIT V - APPLICATIONS
Unit – 1 Introduction to Soft
Computing, Artificial Neural
Networks
 Unit – 1
• Introduction to Soft Computing: Introduction to soft computing -
soft computing vs. hard computing - various types of soft computing
techniques - applications of soft computing.
• Artificial Neural Networks: Neuron - Nerve structure and synapse -
Artificial Neuron and its model - activation functions - Neural network
architecture: single layer and multi-layer feed forward networks -
McCulloch-Pitts neuron model - perceptron model - Adaline and
Madaline - multilayer perceptron model - back propagation learning
methods - effect of learning rule co-efficient - back propagation
algorithm - factors affecting back propagation training - applications.
Introduction to Soft
Computing
Computing:
• It means there are certain input and then
there is a procedure by which the input can
be converted into some output.
• The input is called the antecedent and then
output is called the consequence and then
computing is basically mapping.
• f is the function which is responsible to
convert the input (x) to some output (y).
• This is the concept of computing.
• The concept of computing is first time
invented by a mathematician Lotfi A Zadeh.
• He is only termed as LAZ and he basically is
the first person to introduce the concept of
hard computing.
According to LAZ, the hard computing provides
• precise result
• step that is required to solve the problem is
unambiguous
• control action is formally defined by means
of some mathematical formula or some
algorithm.
 Examples of Hard Computing

• Solving numerical problems (Ex: Roots of

polynomials, integration etc.,)

• Searching and Sorting Techniques

• Solving computational geometry problems

• Basically, three types of problem-solving
approach are available.
• They are
– Hard computing
– Soft computing
– Hybrid computing which is the combination of
hard and soft computing.
• The traditional hard computing approach, where an
exact model of the plant under investigation is
available and traditional mathematical methods are
used to solve the problem.
• The soft computing approach where only an
approximate model of the plant may be available,
and the solution depends upon approximate
reasoning techniques.
• Reasoning means that the action of thinking about
something in a logical, sensible way.
• One of the problems in traditional control systems is that complex
plants cannot be accurately described by mathematical models, and
are therefore difficult to control using such existing methods.
• For example, many nonlinear and time-variant plants with large time
delays cannot easily be controlled and stabilized using traditional
techniques. One of the reasons for this difficulty is the lack of an
accurate model that describes the plant.
• Soft computing, as opposite to traditional computing, deals with
approximate models and gives solutions to complex real-life
problems.
• In effect, the role model for soft computing is the human mind.

• The term soft computing was proposed by the inventor of fuzzy

logic, Lotfi A. Zadeh, 1994. He quoted that “Soft computing is a
collection of methodologies that aim to exploit the tolerance for
imprecision and uncertainty to achieve tractability, robustness,
and low solution cost".
• The first thing tolerance for imprecision is
important. This means that the result that is
obtained using the soft computing not
necessarily to be précised and obviously, the
result is uncertain.
• This is because if you solve this problem several
times it may give different result at different
time.
• And is a robustness, means it can tackle any sort
of input noise.
• And very important concept is called the low
solution cost. Some problems if we follow hard
computing then it is computationally expensive.
• However, if we follow soft computing then it is
computationally very cheap; that means, we
can find a solution in real time.
• Figure shows the hard and soft computing
based problem solution principle as
suggested by Gupta and Kulkarni.
 Characteristics of Soft computing

• It does not require any mathematical modeling of

problem solving.
• It may not yield the precise solution.
• Algorithm are adaptive (i.e. it can adjust to the
change of dynamic environment).
• Use some biological inspired methodologies such as
genetics, evolution, Ant's behaviours, particles
swarming, human nervous system, etc.
Examples of Soft Computing
 Hard computing versus soft computing
Hard computing Soft computing
Hard computing is conventional It is tolerant of imprecision,
computing, requires a precisely stated uncertainty and approximation. The
analytic model. role model for soft computing is the
human mind.
It takes lot of computation time. Less computation time.
It is based on binary logic, crisp Soft computing based on fuzzy logic,
systems, numerical analysis. neural nets and probabilistic
reasoning.
Hard computing requires programs to Soft computing can evolve its own
be written. programs.
Uses Two-valued logic. Use Multi valued or fuzzy logic.
Deterministic Stochastic
Requires exact input data, is strictly Can deal with ambiguous and noisy
sequential, produces precise answers. data, allows parallel computations to
yield approximate answers.
 Various Types of soft computing

Some popular algorithm are

 Fuzzy Logic

 Neural Network

 Genetic algorithm
 Applications of Soft Computing
• Image Processing, pattern classification.
• Image Processing and Data Compression
• Automotive Systems and Manufacturing
• Soft Computing to Architecture
• Decision-support Systems
• Soft Computing to Power Systems
• Neuro Fuzzy systems
• Fuzzy Logic Control
• Machine Learning Applications
• Speech and Vision Recognition Systems
• Process Control and So On…
 Hybrid Computing

• It is a combination of the conventional hard computing and emerging soft

computing.
• So, few portion of the problem can be solved using hard computing for which we

have a mathematical formulation for that particular part and then where we

required a precise input.

• And there are some part of the same problem maybe which cannot be solved in

real time for which no good algorithm is available and we also do not required

accurate result some near accurate result is sufficient for us then we can solve

soft computing for that part and then mixing together is basically the hybrid

computing.
• The major three hybrid systems are as
follows:
– Hybrid Fuzzy Logic (FL) Systems
– Hybrid Neural Network (NN) Systems
– Hybrid Evolutionary Algorithm (EA) Systems
 Unit – 1
• Introduction to Soft Computing: Introduction to soft computing -
soft computing vs. hard computing - various types of soft computing
techniques - applications of soft computing.
• Artificial Neural Networks: Neuron - Nerve structure and synapse -
Artificial Neuron and its model - activation functions - Neural network
architecture: single layer and multi-layer feed forward networks -
McCulloch-Pitts neuron model - perceptron model - Adaline and
Madaline - multilayer perceptron model - back propagation learning
methods - effect of learning rule co-efficient - back propagation
algorithm - factors affecting back propagation training - applications.
 Human Brain
• The human is an amazing processor. The most
basic element of the human brain is a specific
type of cell, known as neuron.
• The human brain comprises about 100 billion
neurons.
• Each neuron can connect with up to 2,00,000
other neurons, although 1,000 - 10,000
interconnections are typical.
• Neural Networks are networks of neurons, for example, as
found in real (i.e. biological) brains.
• Artificial neurons are basic approximations of the neurons
found in real brains. They may be physical devices, or
purely mathematical constructs.
• Artificial Neural Networks (ANNs) : An artificial neural
network may be defined as an information-processing
model that is inspired by the way biological nervous
system, such as the brain, process information. This model
tries to replicate only the most basic functions of the brain.
• From a practical point of view, an ANN is just a parallel computational system

consisting of many simple processing elements (neurons) connected together in

a specific way in order to perform a particular task.

• Also, a neural network is a processing device, either an algorithm or an actual

hardware, whose design was inspired by the design and functioning of animal

brains.
• The neural networks have the ability to learn by example, which makes them

very flexible and powerful.

• For neural networks, there is no need to create an algorithm to perform specific

task, that is, there is no need to understand the internal mechanism of that

task.
• These networks are well suited for real time systems because of their fast

response and computational times which are because of their parallel

 Definitions of neural network

• According to the DARPA Neural Network

Study (1988, AFCEA International Press)
... a neural network is a system composed of many
simple processing elements operating in parallel
whose function is determined by network structure,
connection strengths, and the processing performed
at computing elements or nodes.
 Advantages of Neural Network

• Adaptive learning: An ANN is endowed with the ability to learn how to do

tasks based on the data given for training or initial experience.

• Self organization: An ANN can create its own organization or representation of

the information it receives during learning time.

• Real-time operation: ANN computations may be carried out in parallel. Special

hardware devices are being designed and manufactured to take advantage of

this capability of ANNs.

• Fault tolerance via redundant information coding: Partial destruction of a

neural network leads to the corresponding degradation of performance.

However, some network capabilities may be retained even after major

network damaged.
• Neural network can be viewed from a multi-
disciplinary point of view as shown in Figure.
 Application of NN
• Air traffic control.

• Animal behavior, predator/prey relationships and population cycles.

• Data mining, cleaning and validation.

• Direct mail advertisers.

• Echo patterns.

• Employee hiring.

• Machinery control.

• Medical diagnosis.

• Voice recognition.

• Weather prediction.

• Photos and fingerprints.

 Biological Neuron
• A neural network can be defined as a model of reasoning based on
the human brain. The brain consists of a densely interconnected set
of nerve cells, or basic information-processing units, called
neurons.
• A neuron consists of a cell body, soma, a number of fibers called
dendrites, and a single long fiber called the axon.
Synapse

Synapse Dendrites
Axon
Axon

Soma Soma
Dendrites
Synapse
• An artificial neural network consists of a number of
very simple processors, also called neurons, which are
analogous to the biological neurons in the brain.

Analogy between biological and artificial neural networks

Biological Neur al Network Artificial Neur al Network
Soma Neuron
Dendrite Input
Axon Output
Synapse Weight
• When a neuron receives enough electric pulses through
its dendrites, it activates and fires a pulse through its
axon, which is then received by other neurons. In this
way information can propagate through the NN.
• The synaptic connections change throughout the
lifetime of a neuron and the amount of incoming pulses
needed to activate a neuron (the threshold) also change.
This behavior allows the NN to learn
Mathematical Model of Neuron
Comparison Between Biological Neuron and
Artificial Neuron (Brain vs. Computer)

1011
10 15

Allocation for storage to a new process is easy as it

is added just by adjusting the interconnection
strengths.
 Types of Connections/Architectures
• An ANN consists of a set of highly interconnected processing
elements (neurons) such that each processing element output is
found to be connected through weights to the other processing
elements or to itself, delay lead and lag-free are allowed.
• The arrangement of neuron from layers and the connection
pattern formed within and between layers is called the network
architecture.
1. Single Layer Feed forward network/Multi layer Feed forward

2. Single Layer Feed Back(Recurrent)/ Multilayer Recurrent

 Single Layer Feed forward network
When a layer of processing nodes is formed, the input can be connected to
these nodes with various weights, resulting in series of outputs.
 Multi Layer Feed forward network
Formed by intersection by several layers.
Input layer: Receive input, buffers input signal
Output layer: Generates output
Hidden layer: Internal to the network, no direct contact with external
environment, layer may be 0 to n. If number of hidden layer increases then
results in efficient output.
 Single Layer recurrent network
Processing element output can be directed back to the processing element
itself or to the other processing elements or both.
 Multi Layer recurrent network

Processing element output can be

directed back to the nodes in a
preceding layer.
 Learning/Training Process

The main property of an ANN is its capability to learn.

Learning or training is a process by means of which a neural
network adapts itself to a stimulus by making proper
parameter adjustments, resulting in the production of desired
response.
1)Supervised Learning.
2)Unsupervised Learning.
3)Reinforcement Learning.
 Supervised Learning
• The process of learning from the training dataset can be thought of as a
teacher supervising the learning process.
• Correct target is known.

• Example Network: Perceptron, Adaline, Madaline, BPN

• General Example: Text categorization, Face Detection, Signature recognition

 Unsupervised Learning
• Input of similar type are grouped without use of training data. When
new input pattern is applied, neural network gives the output
response indicating the class to which the input belongs.
• Learning process is independent.

General Example: Coins without denomination

Sample Network Example: Kohonen self organizing future map
 Reinforcement Learning

Learning based on critic information (not exact information).

Feedback obtained is evaluative and not instructive. Learning
through experience.
Example: Child learning whether to touch hot cup of coffee or
not.
 Activation Functions

• Activation function is a function which decide whether the neuron

should be activated or not.

• Used to compute the output response of the neuron.

Types:

1. Linear activation function (applicable of single layer network)

2. Non linear activation function (applicable of multi layer network)

If linear function is applied to multi layer network then output remains

same as single layer network.
 Activation Functions

Types of activation function:

1. Identity function

2. Binary step function

3. Bipolar step function

4. Sigmoidal functions
1. Binary sigmoid function

2. Bipolar sigmoid function

5. Ramp function
 Applicable to: Single layer network having binary data

Bipolar Step Function

Applicable to: Single layer network having bipolar data

Sigmoidal Function Applicable to non-linear system and suitable for BPN. It reduce
computational burden
Binary Sigmoidal Function Bipolar Sigmoidal Function

Ramp Function
 Important Terminologies
1.Weight : Represents the strength of synapse connecting input and output neuron.

Positive weight: weights corresponding to excitatory synapse.

Negative weight: weights corresponding to inhibitory synapse.

2. Connection Matrix: weight written in matrix form

3.Bias: Biases are constant and it is an additional input into the next layer. It act as a weight

whose activation is always one, which improves the performance of neural net.

4.Threshold: It determines, based on the inputs, whether the perceptron fires or not.

5.Learning Rate(α): Used to control amount of weight adjustment at each step of training.

Ranges between 0 and 1.

6.Momentum factor(): Used to converge faster even for higher value of α with less overshoot.

Used in BPN.

7.Vigilance Parameter(ρ): Used to control degree of similarity required for patterns to be

assigned to same cluster unit. Used in adaptive resonance theory. Ranges between 0.7 and 1.
 Various Neural Network Model

• McCulloch-Pitts Neuron (Hebb Rule)

• Preceptron Model (Delta Rule)
• Back Propagation Network
• Adaline, Madaline
• Hopfield Network
Problem
Problem
 McCulloch-Pitts (MCP or MP Neuron)
Architecture

McCulloch (neuroscientist) and Pitts (logician) proposed

a highly simplified computational model of the neuron
(1943)
 McCulloch-Pitts (MCP) Architecture

If the weight on a path is positive the path is excitatory,

otherwise it is inhibitory
• Activation function is used is either Binary step function
or Bipolar step function

Binary step function Bipolar step function

 McCulloch-Pitts (MCP) Architecture

• It (Activation function) simply classifies the set of inputs into two

different classes.

• Thus the output is binary.

• All excitatory connections into a particular neuron have the same weight,
although different weighted connections can be input to different neurons

• weights of the neuron are set along with the threshold to make the neuron
to perform a simple logic function
 Concept of Linear Separability
• Linear separability, is the concept wherein the
separation of the input space into regions is based
on whether the network response is positive or
negative
• A decision line is drawn to separate the positive
and negative responses
• Decision line is called as decision making line
 Concept of Linear Separability
with two inputs

• Decision boundary is a line

 Concept of Linear Separability
with two inputs
Decision criteria with net input greater than zero (i.e) when
threshold is zero
 Concept of Linear Separability
with two inputs
Decision criteria with net input greater than threshold
 Concept of Linear Separability with two inputs

Consider a network having positive response in the first

quadrant and negative response in all other quadrants (AND
function) with either binary or bipolar data, then the decision
line is drawn separating the positive response region from
the negative response region.
What is Training Data & Test Data?
ADALINE Architecture
 ADALINE Architecture
• Units with linear activation function are called as linear units

• Network with single linear unit is called as ADALINE (Adaptive

Linear Neuron)
• In ADALINE input and output relationship is linear

• ADALINE is a neuron, which receives input from several units

and also from one unit called bias
• Weight and bias are adjustable according to the learning
algorithm
• It can be trained using Delta Rule
 Delta Rule

A kind of supervised learning algorithm. Also known

as the Widrow-Hoff rule, or the Delta rule, or the LMS
rule,
Delta rule updates the weights between the
connections so as to minimize the mean-squared error
between the activation and target value
 Delta Rule
The delta rule for adjusting the weight of i th pattern
(i= 1 to n, where n=total no. of pattern ) is
wi   (t  yin ) xi
wi  weight change
  learning rate
x  input activation vector
t  target output
yin  Output of neron  b  x w
i i
Training algorithm
 Perceptron
• Perceptron networks come under single-layer feed
forward networks and they are also called as simple
Perceptron.

• The Perceptron network consists of three units, namely,

sensory unit (input unit), associator unit (hidden unit
different from hidden layer), response unit (output unit).

• Binary activation function is used in sensory unit

• Goal of Perceptron is to classify the input pattern as a

member or not a member to a particular class
• Can only be used for linearly separable problems
Perceptron
Perceptron Activation function
Perceptron Architecture
Perceptron Training Algorithm
 Perceptron Training Algorithm
4. Calculate the output y by applying activation function

Step 6: Train the network until there is no weight change for all
training pair
The weight W1 = I, W2 = l, b = 1 are the final weights after first input
pattern is presented. The same process is repeated for all the input
patterns. The process can be stopped when all the targets become equal
to the calculated output or when a separating line is obtained using the
final weights for separating the positive responses from negative
responses. Table 2 shows the training of perceptron network until its
target and calculated ourput converge for all the patterns.
It can be easily found that the above straight
line separates the positive response and
negative response region, as shown in Figure 2.
 MADALINE Network
• MADALINE which stands for Multiple Adaptive Linear Neuron
• It is a network which consists of many Adalines in parallel with a
single output unit
• It is an example for multilayer feed-forward architecture
• The weights that are connected from the Adaline layer to
Madaline layer are fixed, positive and posses equal values
• Weights between the input layer and Adaline layer are adjusted
during training process
• The network is trained using LMS algorithm
 MADALINE Network

It consists of "n" units of input layer, "m" units of Adaline layer and
"1" unit of the Madaline layer.
MADALINE Network Training Algorithm
 MADALINE Network
• Each neuron in the Adaline and Madaline layers has a bias of
excitation 1
• The Adaline layer is present between the input layer and the
Madaline (output) layer; hence, the Adaline layer can be considered
a hidden layer
• The use of the hidden layer gives the high computational capability
compared to single-layer nets
• but this complicates the training process to some extent
• It is used in adaptive noise cancellation and adaptive filters
MADALINE Network Training Algorithm
MADALINE Network Training Algorithm
 Back-Propagation Network

• Networks associated with back-propagation learning algorithm are

called as back-propagation networks (BPN’s)
• It is an example for multi-layer feed-forward network
• It consists of input layer, one or more hidden layer and an output
layer
• It consists of processing elements (Neurons) with any activation
function which are continuously differentiable
Back-Propagation Network
 Back-Propagation Network
Input signals
1
x1
1 y1
1
2
x2 2 y2
2

i wij j wjk
xi k yk

m
n l yl
xn
Input Hidden Output
layer layer layer

Erro r signals
 Back-Propagation Network
• Neurons present in the hidden & output layers have
biases whose activation is always one
• Bias term also acts as weights
• Direction of information flow for feed-forward
phase is shown in figure
• During back propagation phase of learning signals
are sent in reverse direction
 Back-Propagation Network
• Training of BPN is done in three stages.
• Feed-forward phase (Feed-forward of input
pattern & calculation of output)
• Back-propagation of Error phase
• Weight updation phase
 Feed forward Training Phase

Calculate the output of hidden unit by applying its

activation function

Send the output of hidden unit to input of output layer

units
Feed forward Training Phase
 Back-propagation of Error
Step -6: Each output unit yk (k=1 to m) receives a target pattern to
the input training pattern and computes the error correction term

On the basis of the calculated error correction term, update the

change in weights and bias:
Weight Updation Phase