1. Neural Networks and their Application in
Catalysis
Anshumaan Bajpai
04/04/2016
1

2. Outline
 Brief History of Artificial Neural Networks (ANNs)
 ANN Basics
 Behler-Parrinello application of Neural Networks to
Potential Energy Surfaces
2

3. Artificial Neural Networks (ANN)
 ANNs are an old idea
 Popular in late 80s and early 90s
 Fell out of favor in late 90s as their training was too slow
 With the computational power of modern computers , they
are the state of art modeling technique for many applications
3

4. Non linear functions
4
X1
X2
X1
X2
Linear Decision boundary Non-Linear Decision boundary
However, if we have variables that are in the order 100s or 1000s then scanning
through all possible higher order polynomial becomes prohibitive

5. Computer vision
5
A simple 20 pixel X 20 pixel image: 400 variables
What we see !! What computers see!!

6. Mimic the Brain
6
: Learns to hear
 Same piece of physical brain
tissue that can process sight,
sound or touch
 May be there is a single
algorithm that can be applied
across multiple fields

7. Mimic the Neuron
7
Sigmoidal Activation function
X1
X2
X3
X0
Ɵ0
Ɵ1
Ɵ2
Ɵ3
Bias unit = 1
ℎ 𝜃 𝑥 =
1
1 + 𝑒− 𝜃 𝑖 𝑥 𝑖
ℎ 𝜃 𝑥
𝜃𝑖 𝑥𝑖

8. Network of Neurons
8
X1
X2
X3
𝑎1
(2)
𝑎2
(2)
𝑎3
(2)
𝑎1
(2)
= 𝑔(𝜃10
1
𝑥0 + 𝜃11
1
𝑥1 + 𝜃12
1
𝑥2 + 𝜃13
1
𝑥3)
𝑎2
(2)
= 𝑔(𝜃20
1
𝑥0 + 𝜃21
1
𝑥1 + 𝜃22
1
𝑥2 + 𝜃23
1
𝑥3)
𝑎3
(2)
= 𝑔(𝜃30
1
𝑥0 + 𝜃31
1
𝑥1 + 𝜃32
1
𝑥2 + 𝜃33
1
𝑥3)
ℎ 𝜃 𝑥 = 𝑎1
(3)
= 𝑔(𝜃10
2
𝑎0
(2)
+ 𝜃11
2
𝑎1
(2)
+ 𝜃12
2
𝑎2
(2)
+ 𝜃13
2
𝑎3
(2)
)
𝑔 𝑧 =
1
1 + 𝑒−𝑧
ℎ 𝜃 𝑥
𝑎𝑖
(𝑗)
= “activation” of unit i in layer j
𝜃(𝑗) = matrix of weights controlling
function mapping from layer j to layer j+1
Hidden layer
Node

9. Cluster Expansion substitute
9
 ANNs are very effective in dealing
with images
 Renders naturally to any kind of
system with a grid of numbers
 Simple surfaces (111, 100, etc.) can
easily be represented as a 2D grid
of adsorbate occupancies

10. Major Issue
• Supercells with all different sizes and shapes
10

11. Behler Parrinello approach
11
 Defined NN for each element in the
system
 Inputs to these Element based NN
are a set of symmetry functions
 These symmetry functions uniquely
define the chemical environment of
that particular atom
 Model is trained on the sum of the
output of all these neural networks
Behler J. Int. J. Quantum Chem., 2015, 115, 1032–1050

12. Symmetry function
12
 rotational and translational invariant
 invariant with respect to the permutation of atoms of the
same element
 provides a unique description of the atomic positions
 constant number of function values independent of the
number of atoms in the cutoff spheres
𝐺𝑖
1
=
𝑗=1
𝑁 𝑎𝑡𝑜𝑚
𝑓𝑐(𝑅𝑖𝑗)
𝐺𝑖
2
=
𝑗=1
𝑁 𝑎𝑡𝑜𝑚
𝑒−𝜂(𝑅 𝑖𝑗− 𝑅 𝑠)2
. 𝑓𝑐(𝑅𝑖𝑗)
𝑓𝑐 𝑅𝑖𝑗 =
0.5 ∗ 𝑐𝑜𝑠
𝜋𝑅𝑖𝑗
𝑅 𝑐
+ 1 𝑓𝑜𝑟 𝑅𝑖𝑗 ≤ 𝑅 𝑐
0.0 𝑓𝑜𝑟 𝑅𝑖𝑗 ≥ 𝑅 𝑐
Behler J. Int. J. Quantum Chem., 2015, 115, 1032–1050

13. Summary
 Pros:
– Universal PES approximator
– Available as a python library written by Andrew Peterson and
Alireza Khorshidi from Brown University
– Every geometric step can be used as training data
– Integrates with Atomic Simulation Environment (ASE)
 Cons:
– Difficult to make inferences
– Needs a large training set
13

14. Thank You
14