1. Neuromorphic solutions
for autonomous land
and aerial vehicles
Massimiliano Versace
CEO, Neurala Inc.
Director, Boston University Neuromorphics Lab
www.neurala.com
Copyright 2013 Neurala, Inc.
nl.bu.edu
●
Proprietary and Confidential
●
versace@neurala.com
www.neurala.com ● +1-617-418-6161
1

2. To produce virtual and robotic agents
that autonomously learn to perform
useful tasks
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
2

3. Why are not robots everywhere?
limited intelligence
1:1 ratio robot/
human operator
barriers to adoption
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
3

4. Three ingredients for success
Smart mind
Powerful brain
Copyright 2013 Neurala, Inc.
Inexpensive body
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
4

5. The body
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
5

6. Low cost of sensors, new actuators
navigation
accelerometer
gyroscope
magnetometer
front and rear cameras
NFC
barometer
speaker
microphone
proximity
light sensor
Bluetooth
GPS
WiFi + cellular
humidity
temperature
Copyright 2013 Neurala, Inc.
vision
audition
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
6

7. Three ingredients for success
Smart mind
Powerful brain
Copyright 2013 Neurala, Inc.
Inexpensive body
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
7

8. Many robots, many minds
R&D $$$
$$ensors
Copyright 2013 Neurala, Inc.
R&D $$
$ensors
●
Proprietary and Confidential
●
R&D $
www.neurala.com ● +1-617-418-6161
8

9. One mind for autonomous robots?
learning
parallel
processing
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
9

10. Why do we LOVE this mess??!!
Neuromorphic = of the form of neurons, brains
Even in apparently simple tasks, many brain
functions work in synergy to help us solve
otherwise VERY HARD PROBLEMS
The brain does not work in silos of competences.
E.g., as you move around in the world, vision helps navigation (e.g., landmark
recognition, obstacle avoidance), navigation helps vision (what to expect in one
environment vs. another).
Ever thought why you can’t recognize the teller of your grocery at the movie theater?
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
10

11. “Stovepiped” sensemakig
Rock detector
Sky detector & cloud detector
Dust devil detector
Onboard Autonomous Science Investigation System for Opportunistic Rover Science, Castano et al., 2007
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
11

12. Extremely un-stovepiped sensemaking!
scene recognition
environment map
with objects
scenes help
define objects
map helps to find objects
object recognition
object help
define scenes
objects help
orient
(e.g. Citgo)
to: obstacle
avoidance
segmentation
optic flow
stereo & depth
coarse
view
bias where to look
(data acquisition)
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
12

13. Feedforward and feedback processing
One key aspect of neuromorphic computing is feedback processing
Feedback happens all over the brain
Brain is a massively recurrent architecture
E.g. in the visual system, for each output
fiber from thalamus to the visual cortex, the
thalamus receives ~10 feedback fibers
10X
What is this feedback for?
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
13

14. Mark coming to the conference room
New speakers (Auditory)
Desk 0.5 cm lower
(Proprioceptive)
Leave your desk..
New post-it (Visual) ..walk to the conference room.. Different door knob (Tactile)
Chanel #5 smell
..sit in the conference room
(Olfactory)
Feedback information flow:
Feedforward information flow:
- leave the desk for theare MATCHING memory-based expectations with sensory
At every moment, you talk
- walk down the corridor to the conference room
(and other cortical) INPUT
- find the conference room, find a free spot as close to the exit as humanely
EXPECTATION
EXPECTATION
EXPECTATION
EXPECTATION
EXPECTATION
possible
INPUT
INPUT
Copyright 2013 Neurala, Inc.
INPUT
●
Proprietary and Confidential
INPUT
●
INPUT
www.neurala.com ● +1-617-418-6161
14

15. Feedback focuses on anomalous information
The president of the United States of America
&
?
?
Match memory with current input
Automatically and effortlessly focus on crucial differences
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
15

16. Predictive feedback is ubiquitous in the brain
The sentence “eel is on the ...” is preceded by white noise
orange
wagon
wheel is on the wagon
shoe
eel is on the ...
peel is on the orange
heel is on the shoe
Percept may be determined by the sound that one expects to hear in auditory
context based on previous language experiences
Expectations amplify consistent components of the white noise, while
suppressing inconsistent components
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
16

17. Adaptive Resonance Theory
MATCH
ADAPTIVE RESONANCE THEORY
(ART)
ATTENTIONAL
SUBSYSTEM
F2
ORIENTING
SUBSYSTEM
Σ
Σ
Y
RESET
y
+
Math behind neuromorphic systems?
+
𝑑𝑦 𝑗
= −𝐴𝑦 𝑗 + (𝐵 − 𝑦 𝑗 )
𝑑𝑡
+
F1
-
MISMATCH
-
X
+
MATCHING
CRITERION
Σ
w
𝑛
𝑤 𝑖,𝑗 𝑥 𝑖 − (𝑦 𝑗 + 𝐶)
𝑖=0
𝑦𝑘
𝑘≠𝑗
x
𝑑𝑤 𝑖,𝑗
= −𝐷𝑦 𝑗 𝑤 𝑖,𝑗 + 𝐸𝑥 𝑖 𝑦 𝑗
𝑑𝑡
+
Copyright 2013 Neurala, Inc.
𝑛
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
17

18. With Mark Motter, NASA Langle
NASA
STTR
Phase II
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
18

19. One algorithm for all robots
~32M neurons, ~13B synapses
Sensory
Classification
Prefrontal
Cortex
Binding What
& Where
SelfLocalization
Unclear
Curiosity
Drive
Unclear
Multiple Areas
Complex
Cells
V2
Oriented
Edges
V1
Luminance
LGN
PreProcessing
Retina
Reverse
Spread
Prefrontal Ctx
Forward
Spread
CA3
Color
Opponency
LGN?
Reward
Signal
Basal Ganglia
Dentate Gyrus
Saliency
Map
Posterior PC
Active Goal
Location
Prefrontal Ctx
Forward
Spread Trigger
IT
Reward
Location Map
Unclear
Current
Location
Enthorinal Ctx
PositionInvariant Cells
Goal
Selection Map
Unclear
Desired
Next Location
CA1
Path-Finding
Reset
Lack of
Comfort Drive
Unclear
Motivation
Desired
Heading
Cingulate Ctx
Motor Turn
Command
Motor Cortex
Medial Septum
Navigation
Motor Run
Command
Motor Cortex
Virtual Environment or Robotic Platform
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
19

20. Example application: MoNETA
Morris water maze
Goal
Selection
map
MoNETA’s visual input
1st run
4th run
6th run
Brain diagram
8th run
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
Reverse
spread
www.neurala.com ● +1-617-418-6161
Familiarity
map
Obstacle
map
20

21. One algorithm for all robots
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
21

22. Simulated Mars Rover
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
22

23. Feedback improves heading estimation
fusion
(1)
(2)
Vestibular
gyro &
accel.
Source
Vestibular
Visual
Motor
Integrated ring
Copyright 2013 Neurala, Inc.
Average error (deg)
62.8
35.9
29.9
14.9
●
Proprietary and Confidential
●
Visual
Motor
optic flow
wheel
velocity
Avg. error rate (deg/hr)
1.63
0.916
0.983
0.111
www.neurala.com ● +1-617-418-6161
23

24. How should robots “see”?
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
24

25. Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
25

26. Biological vision is:
Selective (non-uniform sampling)
Multimodal (focus vision on speaker)
Predictive (look where the ball is going)
•
computational savings
(up to 30x documented vs. uniformly
sampled images)
•
•
Traver and
Bernardino (2010)
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
conformal mapping
some invariance to changes in scale
and rotation
www.neurala.com ● +1-617-418-6161
26

27. Artificial visual system
object/scene
exogenous
spatial attention
label
cognitive
biases
WHAT
WHERE
learning
control
what
surface
representations
where
Visual System
Preprocessing
eye
movements
Copyright 2013 Neurala, Inc.
●
input
image
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
27

28. Learning to classify Martian rocks
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
28

29. Expanding the visual system
Temporal continuity:
“how do you know that
the thing you are
looking at now is the
same you look at 1
second ago?”
Upcoming development under the STTR Phase II program:
1) Depth processing, optic-flow aided segmentation, scene “gists”
2) Implementation on robotic platforms
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
29

30. Robotic applications
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
30

31. One brain for all robots: UAVs?
Video courtesy of Mark Motter
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
31

32. Optic flow as a collision clue
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
32

33. Sense and Avoid from Flow Expansion
SAFE model
with Mark Motter, NASA Langley, CIF 2012-13
past environment (s) - action (a) pair
reward/punishment
learning rate
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
value current state/action
random action
33

34. Obstacle avoidance via optic flow learning
Collision rate with other UAVs in collision path
Cost of crashing ~= cost of deviating course
Cost of crashing >> cost of deviating course
Each point = MAvg over 200 trials
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
34

35. Obstacle avoidance via optic flow learning
Rate at which the UAV turns off-course in the
absence of a colliding UAV
Cost of crashing ~= cost of deviating course
Cost of crashing >> cost of deviating course
Each point = MAvg over 200 trials
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
35

36. Obstacle avoidance via optic flow learning
Rate at which the UAV turns off-course
after a UAV has passed
Cost of crashing ~= cost of deviating course
Cost of crashing >> cost of deviating course
Each point = MAvg over 200 trials
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
36

37. Identifying other UAVs: virtual environment
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
37

38. Identifying other UAVs: real video
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
38

39. NASA STTR Phase II timeline
1a) Develop navigation and space variant vision on land robot;
1b) Develop collision avoidance for UAV;
2) Merge 1 & 2 on UAV;
How do pilots look at things?
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
39

40. Three ingredients for success
Smart mind
Powerful brain
Copyright 2013 Neurala, Inc.
Inexpensive body
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
40

41. The brain
=
+
100 billion neurons
250 trillion synapses
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
41

42. Brain power
+
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
42

43. Brains, GPUs, multicore CPUs
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
43

44. Videogames and smartphones
+
=
Graphic Processing Units
(GPUs), now in mobile
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
44

45. Parallel vs. serial
GPU
CPU
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
45

46. digital
48?
25?
12?
2017
2016
4
6
24M
2015
0.65V
2014
12M
2013
0.78V
2012
2018
“fancy”
8M
0.87V
2012
2023
2017
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
# neurons/cm2, power supply
GFLOPS per Watt (GPU) &
# mobile CPU cores
Computing power roadmap
●
www.neurala.com ● +1-617-418-6161
46

47. Mouse brain
# neurons/cm2, power supply
2018
3 cm2
2015
5V
6 cm2
28 V
2012
2023
2017
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
47

48. Human brain
# neurons/cm2, power supply
2018
35 m2
2015
2 kV
70 m2
5 kV
2012
2023
2017
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
48

49. Faster than we think?
The Human Brain Project
$100M/yr.
2013-23
BRAIN Initiative,
projected $300M/yr.
10 yrs.
2012
2023
2017
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
49

50. Not only academia!
The NPU will
"not only mimic human-like perception
but also have the ability
to learn how biological brains do”
* See also IBM, Intel
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
50

51. Towards autonomous machines
Smart mind
Powerful brain
Copyright 2013 Neurala, Inc.
Inexpensive body
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
51

52. Outreach: Boston Museum of Science
1.5 million visitors: the most visited cultural attraction in Boston
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
52

53. NASA?...no, MASA!
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
53

54. ~1800 visitors used the app
Snow inches by day - 2014
1.2
1
0.8
0.6
0.4
0.2
0
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
54

55. Neurala
in partnership with
Massimiliano Versace
CEO, Neurala Inc.
www.neurala.com
versace@neurala.com
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
55

56. Where pathway
To What
Pathway
Reset
Shroud
Fit Shroud
(log-polar)
(log-polar)
Hot Spot
(log-polar)
Hot Spot
(retinal)
Early Vision Processing
Working
memory
(Linear)
Working
memory
Eye mov.
Contours
Diffusion
Attention Shroud
Working Memory
Eye/Head Movement
(head cent.)
(Linear)
Working
memory
Eye/head
mov.
kernel
(Linear)
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
Edges
Input
(log-polar)
www.neurala.com ● +1-617-418-6161
56
56

57. Where pathway
log-polar input representation
reset signal (different object)
shroud forming (bottom)
and fully formed (top)
hot-spot working memory
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
new hot spot
57
(where the eye will look next) 57
www.neurala.com +1-617-418-6161
●

58. What pathway
Associative learning
Core of “what”
pathway
ART learning
Recurrent excitation
Spatial shroud
exists
Where
reset
Inhibition
From external
source
Top-down priming
Teacher
Name Category
Object Category
Boundary/
completed
Shroud from
Where
pathway
From “where”
pathway
View Category
Copyright 2013 Neurala, Inc.
●
Generates a name for the foveated object
(unsupervised). Can also take top down
name assigned by external teacher
(supervised)
Groups multiple views of a single object
into an object category. Reset of object
category is gated by Where signal
Groups similar log-polar views
of same object into view categories
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
58

59. What pathway
top-down
teaching signal
bottom-up
activated name
category
view-specific neuron weights
learning multiple views
of the same digit
current input
viewspecific
neurons
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
Confidential
www.neurala.com +1-617-418-6161
Neurala – AFRL site visit,
●
59
59
59

60. Semantic influence on data collection
?
initial
hypothesis
new instance
(ambiguous view)
5
invariance
0
1
“heat maps”
0
5
0
1
2 … 5
2
3
4
learning
5
Where to look next
to maximize
discrimination
6
“heat maps”
7
8
9
Few presentations
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
60

61. Obstacle avoidance via optic flow learning
Collision rate with otherwise “missed” UAVs
Cost of crashing ~= cost of deviating course
Cost of crashing >> cost of deviating course
Each point = MAvg over 200 trials
Copyright 2013 Neurala, Inc.
●
Proprietary and Confidential
●
www.neurala.com ● +1-617-418-6161
61