2. •This slides adapted from many related
slides, papers, & books.
•Special thanks to:
James Kennedy & Russell Eberhart
Maurice Clerc
Riccardo Poli, Tim Blackwell, Andry Pinto, Hugo Alves
Inês Domingues, Luís Rocha, Susana Cruz
Jaco F. Schutte
Matthew Settles
Satyobroto Talukder
and many more
5. Optimization determines the best-suited solution to
a problem under given circumstances.
Non Linear Optimization problems are generally
very difficult to solve.
Linear/Non Linear problems are equivalent with
Linear/Non Linear function.
Optimization
7. Please find the maxima/minima of the following
functions:
– f(x)=x2
+2
– h(t)=3 + 14t − 5t2
– g(y)=5y3
+ 2x2
− 3x
How do we know it is a minimum or maximum?
Clue: use derivatives & math tools.
Case study
11. Base on problem charecteristics:
Unconstrained Optimization.
Constrained Optimization.
Dynamic Optimization.
Optimization
12. Many optimization problems place no restrictions on
the values of that can be assigned to variables of
the problem.
Unconstrained Optimization
13. The process of optimizing an objective function with
respect to some variables in the presence of
constraints on those variables.
Constrained Optimization
14. Many optimization problems have objective
functions that change over time and such
changes in objective function cause changes in
the position of optima.
Dynamic Optimization
15. There are two types of optimization techniques.
Global optimization technique
Local optimization technique
Optimization Techniques
16. Global optimization technique seek to find a global
minimum or lowest function value and its
corresponding global minimizer
Global minimizer (X*)
Global Opt. Technique
17. Local optimization technique try to find a local
minimum and its corresponding local minimizer
Local minimizer (XL
*)
Local Opt. Technique
19.
For a known (differentiable) function f , calculus
can fairly easily provide us with the minima and
maxima of f .
However, in real-life optimization tasks, this
objective function f is often not directly known.
The objective function is a “black box”.
Real-Life
21. Inspired from the nature social behavior and
dynamic movements with communications of
insects, birds and fish
22. In 1986, Craig Reynolds (a biologist) studied the
flocking behavior of birds.
He described this process in 3 simple behaviors:
Separation
avoid crowding local
flockmates
(neighbors)
Alignment
move towards the
average heading of
local flockmates
(neighbors)
Cohesion
move toward the
average position of
local flockmates
(neighbors)
23. In 1990, Heppner and Grenander: research of bird
flocks searching for corn.
In 1995, James Kennedy (a social psychologist) &
Russell Eberhart (an electrical engineer):
influenced by Heppner and Grenander’s work
developed a powerful optimization method Particle
Swarm Optimization (PSO).
I believe there are many more researcher... :-)
24.
Kennedy and Eberhart first introduce The
Particle Swarm Optimization (PSO) algorithm for
a solution to the complex non-linear optimization
problem by imitating the behavior of bird flocks.
The Particle Swarm Optimization (PSO)
algorithm is a multi-agent parallel search
technique which maintains a swarm of particles
and each particle represents a potential solution
in the swarm.
PSO Origins
25.
All particles fly through a multidimensional
search space where each particle is adjusting its
position according to its own experience and that
of neighbors.
Basic Idea
Each particle adjusts its travelling
speed dynamically corresponding to
the flying experiences of itself and its
neighbors.
27.
Basically, there are two type of PSO algorithm:
– + Local Best (lbest) PSO.
– + Global Best (gbest)
They have been developed which differ in the
size of their neighborhoods.
PSO Algorithm
28. The local best or personal best PSO (lbest/pbest
PSO) method only allows each particle to be
influenced by the best-fit particle chosen from
its neighborhood.
The global best PSO (or gbest PSO) is a method
where the position of each particle is influenced
by the best-fit particle in the entire swarm.
Local vs Global
29.
The larger particle interconnectivity of the
gbest PSO, sometimes it converges faster than
the lbest PSO.
Another is due to the larger diversity of the
lbest PSO, it is less susceptible to being
trapped in local minima.
Local vs Global
30. Local best or personal best PSO (lbest/pbest
PSO) use ring social topology/structure.
The global best PSO (or gbest PSO) use star
social network topology/structure.
ring social topology star social topology
Local vs Global
33. The PSO algorithm consists of just three steps :
1. Evaluate the fitness of each particle.
2. Update individual and global best fitnesses
and positions.
3. Update velocity and position of each particle.
which are repeated until some stopping condition
is met.
PSO Algorithm
36.
A uniform distribution (a rectangular
distribution), is a distribution where the
probability of occurrence is the same for all
values. It has constant probability.
For instance, if a die is thrown, then the
probability of obtaining any one of the six
possible outcomes is 1/6.
Uniform Distribution
40. 1. Number of particles usually between 20-60 or 10-
50.
2. C1 is the importance of personal best value.
3. C2 is the importance of neighborhood best value
Usually C1 + C2 = 4 or C1 = C2 = 2 (empirically chosen value). Wrong
initialization of C1 and C2 may result in divergent or cyclic behavior
4. If velocity is too low → algorithm too slow.
5. If velocity is too high → algorithm too unstable.
Charecteristic of PSO
41. ●6. When C1,= C2 = 0 then all particles continue flying
at their current speed.
7.When C1 > 0 and C2 = 0 then all particles are
independent.
8. C2 > 0 and C1 = 0 then all particles are attracked
to single point (i.e Gbest).
9. When C1 = C2 then all particles attracked towards
the average Pbest and Gbest.
Charecteristic of PSO
42. ●10. When C1 >> C2 each particle is more strongly
influenced by its Pbest position, resulting in excessive
wandering.
11.When C2 >> C1 and C2 = 0 then all particles are
much more influenced by the global best position,
which causes all particles to run prematurely to the
optima.
Charecteristic of PSO
44. + Intensification:
explores the previous solutions, finds the best
solution of a given region
+ Diversification:
searches new solutions, finds the regions with
potentially the best solutions
Charecteristic of PSO
45. Advantages
• Insensitive to scaling of design variables
• Simple implementation
• Easily parallelized for concurrent processing
• Derivative free
• Very few algorithm parameters
• Very efficient global search algorithm
Disadvantages
• Tendency to a fast and premature convergence in mid
optimum points
• Slow convergence in refined search stage (weak local search
ability)
Charecteristic of PSO
46. Please find the maxima/minima of the function:
– f(x)= -x2
+5x +20 ; -10 <= x <= 10
– Use 9 particles :
– X1
= -9,6 X2
= -6
– X3
= -2,6 X4
= -1,1
– X5
= 0,6 X6
= 2,3
– X7
= 2,8 X8
= 8,3 X9
= 10
–
Case study
48. Several approaches
• 2-D Otsu PSO
• Active Target PSO
• Adaptive PSO
• Adaptive Mutation PSO
• Adaptive PSO Guided by Acceleration Information
• Attractive Repulsive Particle Swarm Optimization
• Binary PSO
• Cooperative Multiple PSO
• Dynamic and Adjustable PSO
• Extended Particle Swarms
• …
Davoud Sedighizadeh and Ellips Masehian, “Particle Swarm Optimization Methods, Taxonomy and
Applications”. International Journal of Computer Theory and Engineering, Vol. 1, No. 5, December 2009
49. + Particle Swarm Optimization (PSO) and Ant Colony
Optimization (ACO) are part of Swarm Intelligence
(SI).
+ Swarm intelligence (SI) is the collective behavior
of decentralized, self-organized systems, natural or
artificial.
Key Concepts
50. 1. PSO algorithm basically learned from animal’s
activity or behavior to solve optimization problems.
2. Each member of the population is called a particle
and the population is called a swarm.
3. It does not require any gradient information of the
function to be optimized and uses only primitive
mathematical operators.
Key Concepts
51. 4. PSO is well suited to solve the non-linear, non-convex,
continuous, discrete, integer variable type problems.
5. In PSO, each particle flies through the
multidimensional space and adjusts its position in every
step with its own experience and that of peers toward an
optimum solution by the entire swarm.
6. It doesn’t always work well, still has room for
improvement.
Key Concepts
