The document summarizes Jeff Hawkins' presentation on a proposed framework for understanding intelligence and cortical computation. Some key points: - Hawkins proposes that grid cells exist in the neocortex and represent the location of sensory input relative to objects. Each cortical column learns a complete model of objects, including their location spaces. - Objects can be composed of other objects via displacement cells, allowing efficient learning of new combinations without relearning parts. Behaviors can also be represented as sequences of displacements. - This framework provides insights into neuroscience, concepts, limits of intelligence, and has implications for building true artificial intelligence based on distributed object-centric representations.

1. Human Brain Project Summit
Location, Location, Location
A Framework for Intelligence and Cortical Computation
October 15, 2018
Maastricht, Netherlands
Jeff Hawkins
jhawkins@numenta.com

2. “In spite of the accumulation of detailed knowledge,
how the human brain works is still profoundly mysterious”
To understand the brain we need “new ways of thinking about it,” more
experimental data will not be sufficient
“What is conspicuously lacking is a framework of ideas within which to
interpret all these different approaches.” “It is not that most
neurobiologists do not have some general concept what is going on. The
trouble is the concept is not precisely formulated. Touch it and it
crumbles.”
Crick 1979

3. The Human Neocortex
75% of brain
Organ of intelligence
How it works is still a mystery
The most important scientific problem
of all time
Talk Outline
1) Background
2) A Framework for Intelligence and Cortical Computation
3) Implications

5. Regions and Hierarchy
Retina
Simple
features
Complex
features
ObjectRegion 3
Region 2
Region 1
Somatic regions
Visual regions
Auditory regions
Felleman, van Essen, 1991
Region to region connectivity
Complex
Not strictly hierarchical
(40% of all possible connections exist)
Local circuitry
Remarkably similar everywhere
Macaque monkey
Skin
Simple
features
Complex
features
Object
Multi-modal Object

6. Local Cortical Circuits
Dozens of neuron types
Organized in layers
Prototypical projections across layers
Limited horizontal projections
All regions have a motor output
Similar circuits in all regions
L3
L4
L6a
L6b
L5a
L5b
Sense Motor
L2
Cajal, 1899
2.5mm

7. Vernon Mountcastle’s Big Idea
1) All areas of the neocortex look the same because they perform
the same basic function.
2) What makes one region visual and another auditory is what it is
connected to.
3) A small area of cortex, a 1mm2 “cortical column”, is the unit of
replication and contains the common cortical algorithm.
Mountcastle, 1978

8. Q. What Does the Neocortex Do?
A. The neocortex learns a model of the world
- Thousands of objects, how they look, feel, and sound
- Where objects are located relative to other objects
- How objects behave
- Physical and abstract objects
- A Predictive model
Q. How does the neocortex learn this model?

9. Talk Outline
1) Background
2) A Framework for Intelligence and Cortical Computation
3) Implications

10. Thought Experiment

11. L2/3
L4
L6a
Location
relative
to object
Object
A single column learns completes
models of objects by integrating
features and locations over time.
“A Theory of How Columns in the Neocortex Enable Learning the
Structure of the World” (Hawkins, et. al., 2017)
Multiple columns can infer objects in a single
sensation by “voting” on object identity.
?
Sensed
feature

12. Grid Cells
“Grid Cell” neurons in entorhinal cortex represent the location
of the body relative to an environment.
The Big Idea:
Grid cells also exist in the neocortex.
Cortical grid cells represent the location of sensory input
relative to objects.

13. How Grid Cells Represent Location

14. How Grid Cells Represent Location
A grid cell fires at multiple locations as the animal moves.
A grid cell’s activity is updated by copies of motor commands.
Firing locations of two grid cells
Grid cells cannot represent a unique location.
Grid cell “modules” differ by scale and orientation.
Module 1 Module 2 Unique location
Representation of location is unique to position in room and to the room.
Stensola, Solstad, Frøland, Moser, Moser: 2012

15. Entorhinal Cortex
Learns environments
Grid cells represent location of body relative
to room
Location representation is updated by
movement
Each room has a unique “location space”
Room 1
Room 2
Cortical grid cells represent location of sensor
input relative to object
Location representation is updated by
movement
Each object has a unique “location space”
Neocortex
Learns objects

16. L2/3
L4
L6a
Object
Location
relative
to object
Sensed
feature
?Grid Cell
Modules
Lewis et. al., 2018

17. Our Proposal So Far
1) Grid cells exist in every cortical column.
They represent the location of the input to the column
relative to the object being sensed.
2) Each column learns complete models of objects.
3) Objects have their own unique “location space”.
This defines a location-based framework for understanding
the neocortex and suggests solutions other mysteries.

18. Compositional Structure
x
z
y
a
c
b
Cup is a previously learned object.
Logo is a previously learned object.
How can we rapidly and efficiently learn
new object, “cup with logo”, without
relearning cup or logo?
Objects are composed of other objects, arranged in a particular way.
Cup and logo have their own location
spaces.
Cup with logo can be represented by a
single transform (blue arrow) that
converts any location in cup space to an
equivalent location in logo space.

19. Displacement Cells
(proposed)
x
z
y
a
c
b
Location “x” on logot=2
module 1 module 2 module n
Location “a” on cupt=1Grid cells
Displacement cells “Logo on cup” at a
particular position
Hawkins et. al., 2018

20. L2/3
L4
L6a
Object
Location
Sensed
feature
Grid Cell
Modules
Displacement
Cell Modules
L5
Hawkins et. al., 2018

21. Object Behaviors
Object behaviors can be represented and learned as
sequences of displacements.
Displacement N
Displacement A
Hawkins et. al., 2016
(sequence memory)

22. Rethinking Hierarchy
All columns learn models of objects.
If columns observe the same object then
connections between them are useful for
resolving ambiguity.
(Solves “sensor fusion” problem.)
SenseSense
Simple
features
Complex
features
Object
Multi-modal
Object
Classic view “Thousand Brains Theory of Intelligence”
Retina Skin

23. 1) Cortical Columns:
They are far more powerful than currently believed
Columns learn complete models of objects
Including sub-objects and behaviors
Grid cells and displacement cells define location spaces for
objects and their relative positions
2) Neocortex as a Whole:
Composed of thousands of models
Models differ based on their inputs (Mountcastle)
Long range connections allow columns to vote
Francis Crick
We need “new ways of thinking about the brain”
We need “a broad framework in which to interpret” experimental results
Does this Framework Apply to Concepts and Other Forms of Intelligence?

24. Object-centric location signal in sensory regions:
Cells fire only if feature is at object-centric location on object, even in V1 and V2.
(Zhou et al., 2000; Willford & von der Heydt, 2015)
Grid cells in neocortex:
Human neocortex shows grid cell-like signatures (fMRI and single cell recordings)
(Doeller et al., 2010; Jacobs et al. 2013; Constantinescu et al., 2016; )
Sensorimotor prediction in sensory regions:
Cells predict their activity before a movement is completed.
(Duhamel et al., 1992; Nakamura and Colby, 2002; Li and DiCarlo, 2008)
Neocortex evolved from brain areas involved in mapping and navigation:
Six-layer neocortex evolved by stacking 3-layer hippocampus and piriform cortex
(Jarvis et al., 2005; Luzatti, 2015)
Experimental Support for a Location-based Framework
24

25. Implications
1) Neuroscience
2) Theoretical foundation for
- Pedagogy
- Belief and false belief
- Limits of human intelligence
- Diseases of the mind
3) Artificial Intelligence and Robotics

26. True AI requires
- Distributed models (Thousand Brains Theory of Intelligence)
- Each model built using:
Object-centric locations and location spaces
Compositional structure
Learning through movement
- Embodiment: AI and Robotics are not separable
True AI does not have to be human like
- Faster, Larger, Smaller
- Different sensors
- New embodiments, including virtual

27. Implications of a Neocortical Theory
1) Neuroscience
2) Theoretical foundation for
- Pedagogy
- Belief and false belief
- Limits of human intelligence
- Diseases of the mind
3) Artificial Intelligence and Robotics
- Purpose-built brains, e.g. for mathematics or physics
- Virtual brains, e.g. for cyber-security
- Real robotics, for industry, space exploration, and colonization
www.Numenta.com

29. Numenta Team
Subutai Ahmad Marcus Lewis
Thank You
Scott Purdy Luiz Scheinkman
Mirko Klukas