Neural networks are parallel information processing systems inspired by the brain. They can learn from historical data to make classifications and predictions. A neural network consists of interconnected nodes called neurons that receive inputs, perform operations, and output results. By training neural networks on large amounts of labeled data using techniques like backpropagation, they can learn complex patterns to solve problems like image and speech recognition.

1. 5/7/2012
WHAT ARE NEURAL NETWORKS?
 Neural networks are parallel information processing
systems with their architecture inspired by the
structure and functioning of the brain
 A neural network have the ability to learn and
generalize.
 Neural networks can be trained to make
classifications and predictions based on historical data
 Neural nets are included in many data mining
products
1

2. 5/7/2012
 Very popular and effective techniques
 Artificial Neural Network goes by many names, such
as connectionism, parallel distributed processing,
neuro-computing and natural intelligent systems.
 It is abbreviated by ANN
 ANN has a strong similarity to the biological brain.
 It is composed of interconnected elements called
neurons.
3
The Biological Neuron
 Biological neuron receives inputs from other sources,
combines them in some way, performs a generally
nonlinear operation on the result, and then output the
final result.
send Inputs Process the inputs
(eyes, ears)
Turn the processed
inputs into outputs
The electrochemical contact between
neurons
4
2

3. 5/7/2012
The Artificial Neuron
5
Network Layers
 The neurons are
grouped into layers.
 The input layer.
 The output layer.
 Hidden layers between
these two layers.
6
3

4. 5/7/2012
Neural Network Architecture
 In a Neural Network, neurons are grouped into
layers.
 The neurons in each layer are the same type.
 There are different types of Layers.
 The Input layer, consists of neurons that receive
input from the external environment.
 The Output layer, consists of neurons that
communicate to the user or external environment.
 The Hidden layer, consists of neurons that ONLY
communicate with other layers of the network.
 Now that we have a model for an artificial
neuron, we can imagine connecting many of
then together to form an Artificial Neural
Network:
Output layer
Hidden layer
Input layer
4

5. 5/7/2012
Based on the following assumptions:
1. Information processing occurs at many simple
processing elements called neurons.
2. Signals are passed between neurons over
interconnection links.
3. Each interconnection link has an associated
weight.
4. Each neuron applies an activation function to
determine its output signal.
A Neuron
x0
- mk
w0
x1 w1
 f output y
xn wn
Input weight weighted Activation
vector x vector w sum function
 The n-dimensional input vector x is mapped into
variable y by means of the scalar product and a
nonlinear function mapping
5

6. 5/7/2012
What is a neuron?
 a (biological) neuron is a
node that has many inputs
and one output  f
 inputs come from other
neurons
 the inputs are weighted
 weights can be both positive
and negative
 inputs are summed at the
node to produce an
activation value
The Artificial Neuron Model
 In order to simulate neurons on a computer, we
need a mathematical model of this node
 Node j ( a neuron) has n inputs xi , i = 1 to n
 Each input (connection) is associated with a weight
wij
 The neuron includes a bias denoted by bj
 The bias has the effect of increasing or deceasing
the net input
12
6

7. 5/7/2012
 The net input to node j (net result) ,uj , is the sum of
the products of the connection inputs and their
weights: n
u j   w ijxi  b j
i 1
Where uj is called the “activation of the neuron”.
 The output of node j is determined by applying a non-
linear transfer function g to the net input:
x j  g(u j )
 called the “transfer function”.
 A common choice for the transfer function is the
sigmoid:
1
g(u j )  u j
1 e
 The sigmoid has similar non-linear properties to the
transfer function of real neurons:
 It accepts inputs varying from – ∞ to ∞
 bounded below by 0
 bounded above by 1
7

8. 5/7/2012
An Example of multilayer feed
forward neural network
8

9. 5/7/2012
Architecture of ANN
 Feed-Forward Networks
The signals travel one way from input to output
 Feed-Back Networks
The signals travel as loops in the network, the
output of the network is connected to the
input.
17
Learning ANN
 The purpose of the learning function is to
modify the variable connection weights
causes the network to learn the solution to
a problem according to some neural based
algorithm.
 There are tow types of learning
 Supervised (Reinforcement) learning.
 Unsupervised learning.
18
9

10. 5/7/2012
Supervised learning
 Means their exist an external
help or a teacher.
 The teacher may be a training
set of data.
 The target is to minimize the
error between the desired and
the actual output.
 The process take place as
follows:
19
 Presenting input and output data to the network,
this data is often referred to as the “training set”.
 The network processes the inputs and compares
its resulting outputs against the desired outputs.
 Errors are then propagated back through the
system, causing the system to adjust the weights,
which are usually randomly set to begin with.
 This process occurs over and over as a closer
match between the desired and the predicted
output.
20
10

11. 5/7/2012
 When no further learning is necessary, the
weights are typically frozen for the
application.
 There are many algorithms used to implement
the adaptive feedback required to adjust the
weights during training. The most common
technique is called “Back-Propagation”.
Let us suppose that a sufficiently large set of
examples (training set) is available.
Supervised learning:
– The network answer to each input pattern is
directly compared with the desired answer and a
feedback is given to the network to correct possible
errors
11

12. 5/7/2012
Back Propagation (BP)
 It is an improving performance method in training of
multilayered feed forward neural networks.
 Are used to adjust the weights and biases of
networks to minimize the sum squared error of
the network which is given by
1
SSE 
2
 (xt  xt )2
ˆ
 where xt and xt^ are the desired and predicted
output of the tth output node.
23
BP Network – Supervised Training
 Desired output of the training examples
 Error = difference between actual & desired output
 Change weight relative to error size
 Calculate output layer error , then propagate back to
previous layer
 Hidden weights updated
 Improved performance
24
12

13. 5/7/2012
 Probably the most common type of ANN used
today is a multilayer feed forward network
trained using back-propagation (BP)
 Often called a Multilayer Perceptron (MLP)
Unsupervised learning
 The training set consists of
input training patterns but not
with desired outputs.
 This method often referred to
as self-organization.
 Therefore, the network is
trained without benefit of any
teacher.
26
13

14. 5/7/2012
 Applications in Clustering and reducing dimensionality
 Learning may be very slow
 No help from the outside
 No training data, no information available on the
desired output
 Learning by doing
 Used to pick out structure in the input:
 Clustering
 Compression
27
Choosing the network size
 It seems better to start with a small number of
neurons, because:
 learning is faster.
 it is often enough.
 it avoids over-fitting problems.
 If the number of neurons are too much we will get
an over fit.
 In principle, one hidden layer is sufficient to solve
any problem. In practice, it may happen that two
hidden layers with a small number of neurons may
work better (and/or learn faster) than a network
with a single layer.
28
14

15. 5/7/2012
• Too few nodes: Don’t fit the curve very well
• Too many nodes: Over parameterization
• May fit noise
• makes network more difficult to train
The Learning Rate
 The learning rate c, which determines by how
much we change the weights w at each step.
 If c is too small, the algorithm will take a long
time to converge.
Sum-Square Error
Error Surface
Epoch
30
15

16. 5/7/2012
 If c is too large, the network may not be
able to make the fine discriminations
possible with a system that learns more
slowly. The algorithm diverges.
Sum-Square Error
Error Surface
Epoch
31
Multi-Layer Perceptron
Output vector
Errj  O j (1  O j ) Errk w jk
Output nodes k
 j   j  (l) Errj
wij  wij  (l ) Errj Oi
Hidden nodes
Errj  O j (1  O j )(T j  O j )
wij 1
Oj  I
1 e j
Input nodes
I j   wij Oi   j
i
Input vector: xi
16

17. 5/7/2012
Applications of ANNs
 Prediction – weather, stocks, disease, Predicting
financial time series
 Classification – financial risk assessment, image
processing
 Data Association – Text Recognition
 Data Conceptualization – Customer purchasing
habits
 Filtering – Normalizing telephone signals
 Optimization
33
 Diagnosing medical conditions
 Identifying clusters in customer databases
 Identifying fraudulent credit card transactions
 Hand-written character recognition
 and many more….
17

18. 5/7/2012
 Advantages
 Adapt to unknown situations
 Autonomous learning and generalization
 Their most important advantage is in solving
problems that are too complex for conventional
technologies, problems that do not have an
algorithmic solution or for which an algorithmic
solution is too complex to be found.
 Disadvantages
 Not exact
 Large complexity of the network structure
35
Using a neural network for prediction
 Identify input and outputs
 Preprocess inputs - often scale to the range [0,1]
 Choose an ANN architecture
 Train the ANN with a representative set of training
examples (usually using BP)
 Test the ANN with another set of known examples
 often the known data set is divided in to training
and test sets. Cross-validation is a more rigorous
validation procedure.
 Apply the model to unknown input data
18