Slides for the Keynote Talk at SIBGRAPI2015 (Brazil)

1. Nature-Inspired Optimization Algorithms
Xin-She Yang
Middlesex University London
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 1 / 44

2. Live in Nature and Inspired by Nature
At the Gates of Nature and Inspired by Nature
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 2 / 44

3. The Essence of an Algorithm The Essence of an Algorithm
The Essence of an Algorithm
Essence of an Optimization Algorithm
To move to a new, better point x(t+1) from an existing location x(t).
x1
x2
x(t)
x(t+1)
?
Population-based algorithms use multiple, interacting paths.
Different algorithms
Different strategies/approaches in generating these moves!
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 3 / 44

4. The Essence of an Algorithm The Essence of an Algorithm
The Essence of an Algorithm
In essence, an algorithm can be written (mathematically) as
xt+1
= A(xt
, α),
For any given xt, the algorithm will generate a new solution xt+1.
Functional or a Dynamical System?
We can view the above equation as
an iterative functional (with fixed points),
or a dynamical system (with attractors),
or a self-organization system (with organized structures),
or a Markov chain (e.g., Markov chain Monte Carlo).
The behavoir of the system (algorithm) can be controlled by A and α.
For details, please refer to: Xin-She Yang et al., Swarm Intelligence and
Bio-Inspired Computation: Theory and Applications, Elsevier, (2013).
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 4 / 44

5. The Essence of an Algorithm What’s Wrong with Traditional Algorithms?
What’s Wrong with Traditional Algorithms?
Some issues with traditional optimization algorithms such as the
Newton-Raphson Method, pattern search, quadratic programming, etc.
Traditional algorithms are mostly local search, thus cannot guarantee
global optimality (except for linear programming and convex
optimization).
Results often depend on the initial starting points (except linear and
convex problems).
Methods tend to be problem-specific (e.g., k-opt, branch and bound).
Cannot deal with highly nonlinear optimization problems efficiently.
Struggle to cope problems with discontinuity.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 5 / 44

6. The Essence of an Algorithm Nature-Inspired Optimization Algorithms
Nature-Inspired Optimization Algorithms
Almost all nature-inspired metaheuristic optimization algorithms (heuristic
or metaheuristic algorithms, as a global optimizer) strive to
Increase the probability of finding the global optimality (as a global
search tool);
Solve a wider class of problems (treating them as a black-box);
Try to deal with highly nonlinear problems, problems with
discontinuity, and even NP-hard combinatorial problems;
Draw design inspiration from Nature.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 6 / 44

7. Nature-Inspired Algorithms Nature-Inspired Optimization Algorithms
Nature-Inspired Optimization Algorithms
Genetic algorithms (1960s/1970s), evolutionary strategy (Rechenberg
& Swefel 1960s), evolutionary programming (Fogel et al. 1960s).
Simulated annealing (Kirkpatrick et al. 1983), Tabu search (Glover
1980s), ant colony optimization (Dorigo 1992), genetic programming
(Koza 1992), particle swarm optimization (Kennedy & Eberhart
1995), differential evolution (Storn & Price 1996/1997), harmony
search (Geem et al. 2001), honeybee algorithm (Nakrani & Tovey
2004), artificial bee colony (Karaboga, 2005).
Firefly algorithm (Yang 2008), cuckoo search (Yang & Deb 2009), bat
algorithm (Yang, 2010), flower pollination algorithm (2012), ...
For details, please refer to:
Xin-She Yang, Nature-Inspired Optimization Algorithms, Elsevier, (2014).
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 7 / 44

8. Swarm Intelligence Seeking Inspiration in Nature: Swarm Intelligence
Seeking Inspiration in Nature: Swarm Intelligence
Swarm Intelligence
Ants, bees, birds, fish ...
Simple rules lead to complex behaviour.
Swarming Starlings
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 8 / 44

9. Swarm Intelligence PSO
PSO
Particle Swarm Optimization (Kennedy and Eberhart, 1995)
xi
g∗
xj
vt+1
i = vt
i + αǫ1(g∗
− xt
i ) + βǫ2(x∗
i − xt
i ),
xt+1
i = xt
i + vt+1
i .
α, β = learning parameters, ǫ1, ǫ2=random numbers.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 9 / 44

10. Swarm Intelligence PSO Demo and Disadvantages
PSO Demo and Disadvantages
The updating equations in PSO form a linear system:


xi
vi


t+1
=


1 1
−(αǫ1 + βǫ2) 1




xi
vi


t
+


0
αǫ1g∗ + βǫ2x∗
i

.
PSO Demo Premature convergence
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 10 / 44

11. Swarm Intelligence Firefly Algorithm
Firefly Algorithm
Firefly Video
Firefly Behaviour and Idealization (Yang, 2008)
Fireflies are unisex and brightness varies with distance.
Less bright ones will be attracted to bright ones.
If no brighter firefly can be seen, a firefly will move randomly.
xt+1
i = xt
i + β0e−γr2
ij (xj − xi ) + α ǫt
i .
⋄ The objective landscape maps to a light-based landscape, and fireflies
swarm into the brightest points/regions.
⋄ There is no g∗, therefore, there is no leader. Also, a highly nonlinear
iterative system, so subdivision into multiswarms is possible.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 11 / 44

12. Swarm Intelligence FA Demo and Advantages
FA Demo and Advantages
Paths of fireflies for 3D Rosenbrock function (Husselmann, 2014).
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 12 / 44

13. Swarm Intelligence Why is FA so efficient?
Why is FA so efficient?
FA Demo Multimodal optimality
Advantages of Firefly Algorithm
Automatically subdivide the whole population into subgroups, and
each subgroup swarms around a local mode/optimum.
Control modes/ranges by varying γ.
Control randomization by tuning parameters such as α.
Suitable for multimodal, nonlinear, global optimization problems.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 13 / 44

14. Swarm Intelligence Variants of Firefly Algorithm
Variants of Firefly Algorithm
More than 2000 papers about firefly algorithm since 2008. Its literature is
dramatically expanding (Fister et al., 2013)
[http://www.sciencedirect.com/science/article/pii/S2210650213000461 ]
Variants for specific applications:
Continuous optimization, mixed integer programming, ...
Discrete firefly algorithm for scheduling, travelling-salesman problem,
combinatorial optimization ...
Image processing and compression ...
Clustering, classifications and feature selection ...
Chaotic firefly algorithm ...
Hybrid firefly algorithm with other algorithms ...
Multiobjective firefly algorithm.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 14 / 44

15. Swarm Intelligence Cuckoo Breeding Behaviour
Cuckoo Breeding Behaviour
Evolutionary Advantages
Dumps eggs in the nests of host birds and let these host birds raise their
chicks, so minimum efforts and maximum success.
Cuckoo Video (BBC)
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 15 / 44

16. Swarm Intelligence Cuckoo Search (Yang and Deb, 2009)
Cuckoo Search (Yang and Deb, 2009)
Cuckoo Behaviour and Idealization
Each cuckoo lays one egg (solution) at a time, and dumps its egg in a
randomly chosen nest.
The best nests with high-quality eggs (solutions) will carry out to the
next generation.
The egg laid by a cuckoo can be discovered by the host bird with a
probability pa and a nest will then be built.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 16 / 44

17. Swarm Intelligence Cuckoo Search
Cuckoo Search
Local random walk:
xt+1
i = xt
i + s ⊗ H(pa − ǫ) ⊗ (xt
j − xt
k).
[xi , xj , xk are 3 different solutions, H(u) is a Heaviside function, ǫ is a
random number drawn from a uniform distribution, and s is the step size.
Global random walk via L´evy flights:
xt+1
i = xt
i + αL(s, λ), L(s, λ) =
λΓ(λ) sin(πλ/2)
π
1
s1+λ
, (s ≫ s0).
Generation of new moves by L´evy flights, random walk and elitism.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 17 / 44

18. Swarm Intelligence
The normal distribution
p(x; µ, σ2
) =
1
σ
√
2π
exp[−
(x − µ)2
2σ2
], −∞ < x < ∞.
Isotropic Random Walks
SN =
N
i=1
Xi = X1 + ... + XN = SN−1 + XN,
s
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 18 / 44

19. Swarm Intelligence L´evy Distribution and L´evy Flights
L´evy Distribution and L´evy Flights
L(s) =
1
π
∞
0
cos(τs)e−α τγ
dτ, (0 < γ ≤ 2),
=
γ
2π
exp −
γ
2(s − µ)
1
(s − µ)3/2
, 0 < µ < s < ∞ (for steps µ > 0).
s
Self-similar, multi-scale, global random walks.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 19 / 44

20. Swarm Intelligence CS Demo: Highly Efficient!
CS Demo: Highly Efficient!
In essence, CS is a multiscale global optimizer.
More than 1000 papers about cuckoo search since 2009. Its literature is
dramatically expanding.
CS Demo Efficient search with a focus
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 20 / 44

21. Swarm Intelligence Bat Algorithm (Yang, 2010)
Bat Algorithm (Yang, 2010)
BBC Video
Microbats use echolocation for hunting
Ultrasonic short pulses as loud as 110dB with a short period of 5 to
20 ms. Frequencies of 25 kHz to 100 kHz.
Speed up pulse-emission rate and increase loudness when homing at a
prey.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 21 / 44

22. Swarm Intelligence Bat Algorithm
Bat Algorithm
Acoustics of bat echolocation
λ =
v
f
∼ 2 mm to 14 mm.
Rules used in the bat algorithm (updating equations):
fi = fmin + (fmax − fmin)β, β ∈ [0, 1],
vt+1
i = vt
i + (xt
i − x∗)fi , xt+1
i = xt
i + vt
i .
x∗=best solution found so far.
Variations of Loudness and Pulse Rate
At+1
i ← αAt
i , α ∈ (0, 1], rt+1
i = r0
i [1 − exp(−γt)].
Such variations are linked with the updating equations by a stochastic
switch (by comparing with a uniformly distributed random number),
controlling the balance between exploration and exploitation.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 22 / 44

23. Swarm Intelligence Other Algorithms
Other Algorithms
Genetic algorithms
Differential evolution
Artificial immune system
Harmony search
Flower pollination algorithm
Memetic algorithm
...
Reviews
Yang et al., Swarm Intelligence and Bio-Inspired Computation:
Theory and Applications, Elsevier, (2013).
Yang, X. S., Nature-Inspired Optimization Algorithms, Elsevier,
(2014).
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 23 / 44

24. Applications Applications
Applications
Design optimization: structural engineering, product design ...
Scheduling, routing and planning: discrete, combinatorial problems ...
Image processing and data Mining: image segmentation, feature
selection ...
Applications in almost all areas (e.g., finance, economics, engineering,
industry, ...)
10 Examples in Science and Engineering
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 24 / 44

25. Applications Pressure Vessel Design Optimization (Example 1)
Pressure Vessel Design Optimization (Example 1)
r
d1
r
L d2
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 25 / 44

26. Applications Pressure Vessel Design
Pressure Vessel Design
This is a well-known test problem for optimization (e.g., see Cagnina et al.
2008) and it can be written as
minimize f (x) = 0.6224d1rL + 1.7781d2r2
+ 3.1661d2
1 L + 19.84d2
1 r,
subject to



g1(x) = −d1 + 0.0193r ≤ 0
g2(x) = −d2 + 0.00954r ≤ 0
g3(x) = −πr2L − 4π
3 r3 + 1296000 ≤ 0
g4(x) = L − 240 ≤ 0.
The simple bounds are
0.0625 ≤ d1, d2 ≤ 99 × 0.0625, 10.0 ≤ r, L ≤ 200.0.
The best solution (Yang, 2010; Gandomi and Yang, 2011)
f∗ = 6059.714, x∗ = (0.8125, 0.4375, 42.0984, 176.6366).
Now we have a set of analytical solutions (Yang et al., 2013).
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 26 / 44

27. Applications Speed Reducer/Gear Box Design (Example 2)
Speed Reducer/Gear Box Design (Example 2)
Mixed-Integer Programming:
Continuous variables and integers.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 27 / 44

28. Applications
f (x1, x2, x3, x4, x5, x6, x7) = 0.7854x1x2
2 (3.3333x2
3 + 14.9334x3 − 43.0934)
−1.508x1(x2
6 + x2
7 ) + 7.4777(x3
6 + x3
7 ) + 0.7854(x4x2
6 + x5x2
7 ),
subject to
g1 = 27
x1x2
2 x3
− 1 ≤ 0, g2 = 397.5
x1x2
2 x2
3
− 1 ≤ 0,
g3 =
1.93x3
4
x2x3d4
1
− 1 ≤ 0, g4 =
1.93x3
5
x2x3d4
2
− 1 ≤ 0,
g5 = 1
110x3
6
(745x4
hx3
)2 + 16.9 × 106 − 1 ≤ 0,
g6 = 1
85x3
7
(745x5
hx3
)2 + 157.5 × 106 − 1 ≤ 0,
g7 = x2x3
40 − 1 ≤ 0, g8 = 5x2
x1
− 1 ≤ 0,
g9 = x1
12x2
− 1 ≤ 0, g10 = 1.5x6+1.9
x4
− 1 ≤ 0,
g11 = 1.1x7+1.9
x5
− 1 ≤ 0.
Simple bounds are 2.6 ≤ x1 ≤ 3.6, 0.7 ≤ h ≤ 0.8, 17 ≤ x3 ≤ 28,
7.3 ≤ x4 ≤ 8.3, 7.8 ≤ x5 ≤ 8.3, 2.9 ≤ x6 ≤ 3.9, and 5.0 ≤ x7 ≤ 5.5. z
must be integers.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 28 / 44

29. Applications Dome Design (Example 3)
Dome Design (Example 3)
120-bar dome: Divided into 7 groups, 120 design elements, about 200 constraints
(Gandomi and Yang 2011; Yang et al. 2012).
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 29 / 44

30. Applications Tower Design (Example 4)
Tower Design (Example 4)
26-storey tower: 942 design elements, 244 nodal links, 59 groups/types, > 4000
nonlinear constraints (Yang et al. 2011; Gandomi & Yang 2012).
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 30 / 44

31. Applications Car Door Design (Example 5)
Car Door Design (Example 5)
Design better, safer and energy-efficient cars
Minimize weight, low crash deflection (< 32 mm), lower impact force (< 4
kN), .... Even a side barrier has 11 design variables!
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 31 / 44

32. Applications Scheduling(Example 6)
Scheduling(Example 6)
MK Marichelvam, T Prabaharan, XS Yang, Improved cuckoo search algorithm for hybrid flow
shop scheduling problems to minimize makespan, Applied Soft Computing, vol. 19, no. 1,pp.
93-101 (2014).
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 32 / 44

33. Applications Soft Testing and Path Generation
Soft Testing and Path Generation
Srivastava, Mallikarjun, Yang (2013). Optimal test sequence generation
using firefly algorithm, Swarm and Evolutionary Computation, 8(2013)
44-53.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 33 / 44

34. Applications Travel Salesman Problem (Example 7)
Travel Salesman Problem (Example 7)
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 34 / 44

35. Applications Cuckoo Search for TSP
Cuckoo Search for TSP
Aziz Ouaarab, Belad Ahiod, Xin-She Yang, Discrete cuckoo search algorithm for the travelling
salesman problem, Neural Computing and Applications, (2013).
http://link.springer.com/article/10.1007/s00521-013-1402-2.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 35 / 44

36. Applications Data Mining: Feature Selection (Example 8)
Data Mining: Feature Selection (Example 8)
Bat algorithm and cuckoo search are very competitive (Papa et al, 2013; 2014).
Also effective at dimension reduction (in terms of number of key features).
Dataset # samples #features #classes
Diabetes 768 8 2
DNA 2000 180 3
Mushrooms 8124 112 2
OPF BA CS FA ICS PSO0
10
20
30
40
50
60
70
80
Accuracy[%]
61 61 59
61 61 6160 61 61 62 6263 62 62 62 61
Mushrooms
Baseline
Binary
Sigmoid
Hyperbolic Tangent
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 36 / 44

37. Applications Image Processing (Example 9)
Image Processing (Example 9)
Satellite images and flooding estimates (Senthilnath & Yang, 2014)
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 37 / 44

38. Applications Satellite images
Satellite images
Three methods are compared, and FA obtained the best results.
Algorithms Optimal Worst Mean
GA 0.8044 0.7528 0.7824
PSO 0.8057 0.8009 0.8051
FA 0.8126 0.8019 0.8076
Also, Senthilnath et al. compared 14 different algorithms for classifications,
and they found that the firefly algorithm seems to be most efficient
among 14 different classification methods.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 38 / 44

39. Applications
See Fong et al. (2014) for details.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 39 / 44

40. Applications Classifications and Clustering (Example 10)
Classifications and Clustering (Example 10)
K-means clustering with nature-inspired algorithms can produce far better
results than K-means methods (Fong, Deb, Yang, Zhang, 2014).
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 40 / 44

41. Applications
The combination of K-means methods with the bat algorithm (and other
algorithms) can indeed enhance the accuracy of clustering.
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 41 / 44

42. Some Open Problems Some Open Problems
Some Open Problems
Mathematical Analysis: A unified mathematical framework (e.g.,
Markov chain theory, dynamical systems) is needed.
Comparison: What are the best/fair performance measures?
Parameter tuning: How to tune the algorithm-dependent
parameters so that an algorithm can achieve the best performance? Is
there a general framework for this?
Scalability: Can the algorithms that work for small-scale problems be
directly applied to large-scale/high-dimensional problems (for
d > 1000) ?
Intelligence: Smart algorithms may be a buzz word, but can truly
intelligent algorithms be developed?
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 42 / 44

43. Thank you :) Books on Nature-Inspired Algorithms
Books on Nature-Inspired Algorithms
Xin-She Yang, Nature-Inspired Optimization Algorithms, Elsevier, (2014).
Xin-She Yang, Cuckoo Search and Firefly Algorithm, Springer, (2014).
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 43 / 44

44. Thank you :) Thank you :) Any questions?
Thank you :) Any questions?
Xin-She Yang et al., Swarm Intelligence and Bio-Inspired Computation:
Theory and Applications, Elsevier, (2013).
Xin-She Yang, Artificial Intelligence, Evolutionary Computing and
Metaheuristics, (2013).
Xin-She Yang Nature-Inspired Optimization Algorithms Middlesex University London 44 / 44