This document discusses active perception in robots and humans. It notes that humans are able to perceive complex, unstructured environments by actively perceiving through mechanisms like saccades and attention. For robots to operate in dangerous environments, they will need to develop similar active perception abilities to cope with perceptual complexity. The document explores issues like where to direct attention, what to remember, and when to stop perceiving and start acting. It also examines how active perception affects learning, representation, and decision-making.

1. Robo$c
Models
of
Ac$ve
Percep$on
Dimitri
Ognibene,
PhD
Laboratory
for
Morphological
Computa:on
and
Learning
(www.thrish.org)

3. To
subs:tute
humans
in
dangerous
jobs
is
one
of
the
main
goals
of
robo:cs
The
ac$ons
in
these
pictures
are
already
possible
for
robots
of
today.
However…..

5. Perceiving
in
these
environments
is
very
complex:
• Unstructured
• Changing
• Many
different
objects
of
different
scales
and
shapes
• Occlusions
• Other
agents
to
perceive
and
coordinate
with
Currently
only
humans
are
able
to
cope
with
such
level
of
perceptual
complexity…
And
humans
perceive
ac$vely…

6. Active Perception
Ognibene
&
Demiris
2013
• Robo:cs
• Neuroscience
• Automa:c
Diagnosis
• Smart
Devices
&
Environments
• Data
mining

7. Foveal
Vision
(What
does
it
mean
to
perceive
ac:vely?)
7

8. Foveal
Vision
(What
does
it
mean
to
perceive
ac:vely?)
Try
to
grasp
an
apple
with
foveal
vision..
Seeing
becomes
like
sampling
and
remembering

9. Foveal
Vision
(What
does
it
mean
to
perceive
ac:vely?)
Try
to
grasp
an
apple
with
foveal
vision..
Seeing
becomes
like
sampling
and
remembering

10. Foveal
Vision
(What
does
it
mean
to
perceive
ac:vely?)
Try
to
grasp
an
apple
with
foveal
vision..
Seeing
becomes
like
sampling
and
remembering

11. Foveal
Vision
(What
does
it
mean
to
perceive
ac:vely?)
Try
to
grasp
an
apple
with
foveal
vision..
Seeing
becomes
like
sampling
and
remembering

12. Foveal
Vision
(What
does
it
mean
to
perceive
ac:vely?)
Try
to
grasp
an
apple
with
foveal
vision..
Seeing
becomes
like
sampling
and
remembering

13. Foveal
Vision
(What
does
it
mean
to
perceive
ac:vely?)
Try
to
grasp
an
apple
with
foveal
vision..
Seeing
becomes
like
sampling
and
remembering

14. Ac:ve
Percep:on
(AP)
Issues*
• Where
to
look?
• What
to
remember?
• When
to
stop
looking
and
start
ac:ng?
– Enough
informa:on?
– Enough
:me?
– Acquired
informa:on
s:ll
valid?
*See
also
The
Frame
Problem

15. Where
to
look?
Use
only
image
sta:s:cs?
Main
limits
of
base
saliency
models
are:
• No
I]
&
Baldi
2010
task
informa:on
• Do
not
consider
limited
field
of
view

16. Where
To
look?
Informa:on
on
Demand
Yarbus 1967 16

17. Where
to
look?
Context
and
task
informa:on
used
to
drive
percep:on
to
the
target
Vogel
&
de
Freitas
2008

18. Unknown
Task
or
Goal
• Task/Goal
depending
on
other
agents’
presence/
goals
• Mul:ple
affordances
required
for
the
task
Ognibene
&
Demiris
IJCAI
2013

19. Ac:ve
Percep:on
and
Mirror
Neurons
Can
Motor
Control
System
predict
others’
19
ac:ons?
• Encode
ac:on
goal
• Abstracts
trajectory
• Needs
percep:ons

20. Human Robot Interaction as a
Distributed Dynamic Event
Ognibene
&
Demiris
2013

21. Predic:ve
Ac:on
Recogni:on
Field
of
view
Effec:ve
Percep:on-­‐Environment
Coupling
is
necessary
for
:mely
Recogni:on
and
Survival
Ognibene
&
Demiris
2013

22. Predic:ve
Ac:on
Recogni:on
Field
of
view
Effec:ve
Percep:on-­‐Environment
Coupling
is
necessary
for
:mely
Recogni:on
and
Survival
Ognibene
&
Demiris
IJCAI
2013

23. Different
hypotheses
of
target
posi:on
Perceive
to
reduce
Field
of
view
Equally
probable,
not
seen
uncertainty
See
also
“Percep:ons
as
hypotheses:
saccades
as
experiments,
Friston
et
al.
2012”
Ognibene
&
Demiris
IJCAI
2013

24. Perceive
to
reduce
Field
of
view
Hand
movement
changes
distribu:on
uncertainty
Ognibene
&
Demiris
IJCAI
2013

25. Field
of
view
Perceive
to
reduce
uncertainty
Saccade
to
target
hypothesis
Ognibene
&
Demiris
IJCAI
2013

26. Field
of
view
Perceive
to
reduce
uncertainty
No
target
at
posi:on
observed
Ognibene
&
Demiris
IJCAI
2013

27. Field
of
view
Perceive
to
reduce
uncertainty
Update
Distribu:on
Ognibene
&
Demiris
2013

28. for each element i an observation oti
which depends on the configuration Info
of Gain
the sensors Percep✓t. :The on
Control
states and for
observations Inten:on
are continuous
variables.
An:cipa:on
every time step the goal of the system is to select the configuration minimise Minimizing
the event
expected uncertainty
uncertainty (condiover :onal
V entropy
(quantified H(v|..))
by entropy ˆ✓t = argmin
p(ot|o0...t1, ✓t)H(V |oo...t, ✓0...t)dot ✓t
Proposed solution. For the recognition of the event and for the selection sensors configuration it is necessary to compute the posterior P(prior distribution P(v, xt
Z
O
1:N) = P(xt
1:N|v)P(v) and the independence
observed event from the sensor configuration P(v|✓) = P(v), the expression of the posterior P(v|ot+1✓t+1) can be derived through the use rule:
P(v|ot+1, ✓t+1) =
P(ot+1|v, ✓t+1)P(v)
P(ot+1|✓t+1)
Ognibene
Demiris
IJCAI
2013

29. O
N(o|Where Info
Gain
Using
Kalman
Filters
|S| is the determinant of a matrix S. The first order Taylor expansion
of P(o|✓) at point ¯ot+1
v results in:
ˆ✓t+1 ⇡ argmin
✓
X
v
P(v)
1
2
ln |St+1
v,✓t+1| + ln
XV
v0
⇣
P (v0)N(¯ov,✓; ¯ot+1
v0,✓t+1, St+1
v0,✓t+1)
⌘#
Expected
entropy
5
for
hypothesis
v
Difference
of
predic:ons
between
the
models

30. Modelling
the
temporal
coupling
of
percep$on
with
external
events
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
of
saccades
on
the
element
performer best
o
target best
:0.1
target not best
Raperformer not best 0
0 5 10 15 20
Time stesp
Gaze
target
during
event
observa:on
Ognibene
Demiris
IJCAI
2013

31. Results
Ognibene
Demiris
IJCAI
2013

32. Results
Ognibene
Demiris
IJCAI
2013

33. Mul:ple
Complex
Simultaneous
Ac:vi:es

34. Hierarchical
Ac:on
Representa:on
to
Represent
Temporal
Structure
Dynamic
Bayes
Network
Probabilis$c
Grammars
Lee,
Ognibene,
Chang
,
Kim,
Demiris
(Submimed)

35. STARE
Spa:o-­‐Temporal
Amen:on
Reloca:on
for
Mul:ple
Structured
Ac:vi:es
Detec:on
Lee,
Ognibene,
Chang
,
Kim,
Demiris
(Submimed)

36. Ac:ve
Percep:on
(AP)
Issues
• Where
to
look?
• What
to
remember?
• When
to
stop
looking
and
start
ac:ng?
– Enough
informa:on?
– Enough
:me?
– Is
acquired
informa:on
s:ll
valid?

37. Ac:ve
Percep:on
Issues
Is
AP
just
useful
to
cope
with
hidden
informa$on?
• Why
has
evolu:on
selected
amen:on
and
reduc:on
of
percep:ve
space
for
many
species?
• Why
does
a
massively
parallel
system,
like
the
brain,
needs
to
use
a
serial
mechanism
like
amen:on?

38. Ac:ve
Percep:on
Issues
• How
are
decision
making
and
planning
affected
by
AP?
How
computa:on
is
affected
by
AP?
• Is
AP
in
the
brain
reflected
by
a
peculiar
kind
of
“ac:ve
processing”?
• How
is
learning
affected
by
AP?
• How
are
representa:ons
affected
by
AP?
• How
can
the
brain
self-­‐organise
to
support
AP?
• How
would
a
dysfunc:on
of
AP
be
manifest?

39. Ac:ve
Percep:on
Issues
• How
are
decision
making
and
planning
affected
by
AP?
How
computa:on
is
affected
by
AP?
• Is
AP
in
the
brain
reflected
by
a
peculiar
kind
of
“ac:ve
processing”?
• How
is
learning
affected
by
AP?
• How
are
representa:ons
affected
by
AP?
• How
can
the
brain
self-­‐organise
to
support
AP?
• How
would
a
dysfunc:on
of
AP
be
manifest?

40. Ac:ve
Percep:on
and
Learning
Percep:on
control
is
strongly
dependent
on
the
task
Learning
a
new
task
may
require
learning
a
new
percep:on
control
policy
40
Ognibene
Baldassare,
IEEE
TAMD,
2014

41. Foveal
Vision
and
Saliency
Map
May
Speed-­‐Up
Learning
of
“Ecological
Tasks”
Ognibene
Baldassare,
IEEE
TAMD,
2014

42. Subjec:ve
and
efficient
representa:ons
42
Agent
has
a
fovea
and
can
see
colors
only
at
the
center
of
its
field
of
view
Agent
is
rewarded
if
it
touched
the
red
block
The
red
block
is
always
on
the
leq
of
the
green
blocks
Green
blocks
are
very
easy
to
find
Blue
blocks
are
randomly
posi:oned
distractors
What
will
be
the
right
ac$on
to
do,
the
right
representa$on
to
learn
for
the
blue
object?
Ognibene
Baldassarre,
IEEE
TAMD,
2014

43. Subjec:ve
and
efficient
representa:ons
43
What
will
be
the
right
ac$on
to
do,
the
right
representa$on
to
learn
for
the
blue
object?
While
Ognibene
Baldassarre,
IEEE
TAMD,
2014
a
random
ac$on
was
expected
due
to
random
posi:on
of
the
blue
block,
the
agent
learns
a
well
organised
representa:on.
It
moves
from
the
blue
block
up,
down
on
the
same
column
or
right.
The
policy
learnt
by
the
agent
for
the
green
and
red
blocks
biased
the
agent
percep$on
of
the
blue
object
making
it
a
landmark
to
find
the
red
object
and
the
agent
behaviour
effec$ve
even
without
memory.

44. Subjec:ve
and
efficient
representa:ons
The
policy
learnt
by
the
agent
for
the
blue
an
red
blocks
biased
the
agent
percep:on
of
the
blue
object
while
making
its
behaviour
effec:ve.
The
agent
starts
usually
from
the
green
object
and
moves
to
elements
in
the
leq
adjacent
column
expec:ng
to
find
the
red
object.
This
leads
to
ignore
the
blue
blocks
that
are
not
in
the
columns
at
the
leq
of
the
green
blocks
(those
inside
the
orange
circle).
Next
picture
shows
the
resul:ng
perceived
structure
of
the
world.
44
Ognibene
Baldassarre,
IEEE
TAMD,
2014

45. Subjec:ve
and
efficient
representa:ons
Perceived
World
biased
by
Ac$ve
Percep$on
The
policy
learnt
by
the
agent
for
the
green
and
red
blocks
biased
the
agent
percep$on
of
the
blue
object
making
it
a
landmark
to
find
the
red
object
and
the
agent
behaviour
effec$ve
even
without
45
memory.
Sequence
of
observa:ons
and
their
frequency
(grey)
aqer
learning
Ognibene
Baldassarre,
IEEE
TAMD,
2014

46. Subjec:ve
and
efficient
representa:ons
46
Ognibene
Baldassarre,
IEEE
TAMD,
2014

47. Representa:ons
Evolu:on
47
G
R
B
Representa:ons
are
not
formed
in
a
uniform
way.
The
system
shows
a
sequen:al
forma:on
of
different
areas
of
ac:vity.
This
may
be
due
to
the
selec:ve
aspect
of
ac:ve
percep:on
which
enables
percep:on
and
change
only
on
a
subset
of
s:muli.
Ognibene
et
al,
SAB
2008

48. Representa:ons
Evolu:on
48
G
R
B
Ognibene
et
al,
SAB
2008

49. Representa:ons
Evolu:on
49
G
R
B
As
representa:ons
are
not
formed
in
a
uniform
way
the
same
is
true
for
the
behaviours
acquired
by
the
agent.
The
sequen:al
forma:on
of
different
areas
of
ac:vity
may
not
only
be
reflected
in
the
behaviours
sequen:ally
acquired
but
also
Ognibene
et
al,
SAB
2008
be
caused
by
the
increasing
capability
of
the
agent
due
to
acquiring
other
behaviours
and
give
place
to
“scaffolding”
supported
by
AP

50. Representa:ons
Evolu:on
50
G
R
B
Ognibene
et
al,
SAB
2008

51. Ac:ve
Percep:on
Issues
• How
are
decision
making
and
planning
affected
by
AP?
How
computa:on
is
affected
by
AP?
• Is
AP
in
the
brain
reflected
by
a
peculiar
kind
of
“ac:ve
processing”?
• How
are
representa:ons
affected
by
AP?
• How
is
learning
affected
by
AP?
• How
can
the
brain
self-­‐organise
to
support
AP?
• How
would
a
dysfunc:on
of
AP
be
manifest?

52. Inten:on
aware
resource
alloca:on
in
3D
Tracking
for
Precision
Manipula:on

53. Ini:al
Improvements
Introduc:on
of
constraints
for
spa:o-­‐
temporal
consistency
and
op:misa:on
to
exploit
GPUs
and
mul:core
CPUs
….
but
STILL
TOO
SLOW

54. Ini:al
Improvements

55. Inten:on
Aware
Resource
Alloca:on
in
3D
Tracking
for
Precision
Manipula:on
Humans
are
able
of
fast
adap:ve
reac:ons
to
unforeseen
events…
which
requires
fast
(maybe
imprecise)
percep:on

56. Inten:on
Aware
Resource
Alloca:on
in
3D
Tracking
for
Precision
Manipula:on
DARWIN Attention
OBJECT REPRESENTATION
Class ID 3D Posture Confidence
Class ID 3D Posture Confidence
Class ID 3D Posture Confidence
Class ID 3D Posture Confidence
Class ID 3D Posture Confidence
3D Pose Estimator
Depth Image
Tracker
Mask Builder
External
Motion Info
Other Object
Masks
Mask
Occlusion
Occlusion
Occlusion
Image
Camera
Image
ID
Image
Image
Confidence
Confidence
Confidence
OUTPUT
Class ID 3D Posture Confidence
2D Object Detector
DARWIN Cognitive
Class ID 3D Posture Confidence Architecture
Rendered
Image
Rendered
IRmeangdeered
Rendered
Image
Image
Rendered
IRmeangdeered
Image
Intentions Predictions
Context
Sensitive
Resource
Allocation
Appearence Based Fast Tracker
Complex
visual
percep:on
system
running
on
parallel
hardware
with
direct
and
indirect
dependencies
between
the
components

57. Inten:on
Aware
Resource
Alloca:on
in
3D
Tracking
for
Precision
Manipula:on
DARWIN Attention
OBJECT REPRESENTATION
Class ID 3D Posture Confidence
Class ID 3D Posture Confidence
Class ID 3D Posture Confidence
Class ID 3D Posture Confidence
Class ID 3D Posture Confidence
3D Pose Estimator
Depth Image
Tracker
Mask Builder
External
Motion Info
Other Object
Masks
Mask
Occlusion
Occlusion
Occlusion
Image
Camera
Image
ID
Image
Image
Confidence
Confidence
Confidence
OUTPUT
Class ID 3D Posture Confidence
2D Object Detector
DARWIN Cognitive
Class ID 3D Posture Confidence Architecture
Rendered
Image
Rendered
IRmeangdeered
Rendered
Image
Image
Rendered
IRmeangdeered
Image
Intentions Predictions
Context
Sensitive
Resource
Allocation
Appearence Based Fast Tracker

58. Inten:on
Aware
Resource
Alloca:on
in
3D
Tracking
for
Precision
Manipula:on
DARWIN Attention
OBJECT REPRESENTATION
Class ID 3D Posture Confidence
Class ID 3D Posture Confidence
Class ID 3D Posture Confidence
Class ID 3D Posture Confidence
Class ID 3D Posture Confidence
3D Pose Estimator
Depth Image
Tracker
Mask Builder
External
Motion Info
Other Object
Masks
Mask
Occlusion
Occlusion
Occlusion
Image
Camera
Image
ID
Image
Image
Confidence
Confidence
Confidence
OUTPUT
Class ID 3D Posture Confidence
2D Object Detector
DARWIN Cognitive
Class ID 3D Posture Confidence Architecture
Rendered
Image
Rendered
IRmeangdeered
Rendered
Image
Image
Rendered
IRmeangdeered
Image
Intentions Predictions
Context
Sensitive
Resource
Allocation
Appearence Based Fast Tracker
Ac$ve
Percep$on
and
Computa$on
to
reduce
uncertainty
• Intrinsic
scene
saliency:
maximise
expected
overall
predictability
(e.g.
an
object
moving
will
make
salient
also
nearby
objects
that
may
occlude
it
or
deviate
it)
• Agent
Inten:on
-­‐
rise
saliency
changing
predic:ons

59. Embodied
Percep:on
and
Tac:le
1. Humans
apply
certain
strategies
to
detect
hard
abnormali:es
in
soq
:ssues
2. Op:mally
chosen
speed
and
load
of
tac:le
probing
will
lead
to
improved
tumour
detec:on
and
bemer
clinical
outcomes
3. Embodied
percep.on
of
environment
should
be
considered
to
define
op:mal
probing
behaviour
Jelizaveta
Konstan:nova
Laboratory
for
Morphological
Computa$on
and
Learning
(Thrish.org
KCL)
Nantachai
Sornkarn
Thrishantha
Nanayakkara
(PI)
Explora:on

60. Embodied
Percep:on
[1]J.
Konstan$nova,
M.
Li,
M.
Gautam,
P.
Dasgupta,
K.
Althoefer
and
T.
Nanayakkara.
“Behavioral
Characteris:cs
of
Manual
Palpa:on
to
Localize
Hard
Nodules
in
Soq
Tissues”,
in
press,
IEEE
Transac$ons
on
Biomedical
Engineering,
2014.
[2]
Nantachai
Sornkarn,
Thrishantha
Nanayakkara,
Mamhew
Howard,
“Internal
Impedance
Control
Helps
Informa:on
Gain
in
Embodied
Percep:on”,
in
IEEE
Interna:onal
Conference
on
Robo:cs
and
Automa:on
(ICRA),
2014

61. Human
Robot
Hap:c
Guidance
Anuradha
Ranasinghe
Thrishantha
Nanayakkara
(PI)
Guiding
agent
can
be
modeled
as
3rd
order
predic:ve
model
using
a
simple
linear
auto-­‐regressive
model
(Arx).
Human
follower
can
be
molded
as
2nd
order
reac:ve
control
policy.
Confidence
of
the
follower
correlates
to
model
virtual
damping
and
can
be
ac$vely
measured
The
guider
can
modulate
the
pulling
force
in
response
to
the
confidence
level
of
the
follower.

62. Ac:ve
Percep:on
Issues
• How
are
decision
making
and
planning
affected
by
AP?
How
computa:on
is
affected
by
AP?
• Is
AP
in
the
brain
reflected
by
a
peculiar
kind
of
“ac:ve
processing”?
• How
are
representa:ons
affected
by
AP?
• How
is
learning
affected
by
AP?
• How
can
the
brain
self-­‐organise
to
support
AP?
• How
would
a
dysfunc:on
of
AP
be
manifest?

63. Predic:ve
Coding
Mumford
1992
Rao
and
Ballard
1999
Friston
2005
Spratling
2008
Hinton
2007
Clark
2013
Figure
from
Feldman
Friston
2010
Hierarchical
Bayesian
Predic:ve
(Genera:ve)
Model
Predic:ons
flow
backward
and
predic:on
errors
forward
(fast
reac:on)
Accumula:on
of
sensory
evidence
reduces
Predic:on
Error
(or
Surprisal)
and
realises
both
Perceptual
inference
and
Learning
in
a
Unified
Framework
Amen:on
can
be
understood
as
inferring
the
level
of
[un]certainty
(c.f.,
Kalman
gain)

64. Ac:ve
Inference
Friston
2003,2010
BBS
Review
by
Clark
2013
Ac:ve
Inference
is
a
generalisa:on
of
Predic:ve
Coding
to
Ac:on
comple:ng
the
Sensorimotor
Loop
Ac:ons
reduce
predic:on
error
by
realising
predic:ons,
e.g.
predicted
propriocep:ve
state
results
in
a
predic:on
error
which
produces
a
reac:on
(e.g.,
reflects)
Innate
priors
and
interac:on
with
the
environment
determine
behaviour
–
no
need
for
norma:ve
quan::es
like
reward
Varia:onal
Free
Energy
allows
to
consider
–
in
a
tractable
(approximate)
analy:cal
form
–
predic:ons
and
predic:on
error
under
uncertainty
Ac:on,
Percep:on,
Learning
and
Planning
are
unified
under
the
same
computa:onal
principle

65. Ac:ve
Inference
and
Ac:ve
Percep:on
Minimising
Predic:on
Error
in
a
trivial
way
may
lead
an
agent
to
get
stuck
in
the
non-­‐
adap:ve
states,
precluding
Explora:ve
Behaviour
P(u! | o!,γ ) =σ (γ ⋅Q(π ))
Qτ (π ) = EQ(oτ |π )[ln P(oτ |m)]
$$$#$$$%
+ EQ(oτ |π )[D[Q(sτ | oτ ,π ) ||Q(sτ |π )]]
$$$$$$#$$$$$$%
Extrinsic
value
Epistemic
value
Friston,
Rigoli,
Ognibene
et
al
(submimed)
Agent
priors
on
behaviour
π
now
contain
an
epistemic/explora:ve
part:
an
agent
will
tend
to
execute
ac:ons
that
reduce
its
uncertainty
about
states
of
the
world
(c.f.,
maximise
informa:on
gain)
Epistemic
value
corresponds
to
the
Bayesian
Surprise.
Empirically
people
tend
to
direct
their
gaze
towards
salient
visual
features
with
high
Bayesian
surprise
(I]
Baldi
2009)

66. Collaborators
Karl
Friston
(UCL)
Hector
Geffner
(UPF)
Thrish
Nanayakkara
(KCL)
Kris
De
Meyer
(KCL)
Giovanni
Pezzulo
(CNR)
Giuseppe
Giglia
(Uni
Pa)
Yiannis
Demiris
(Imperial)
Gianluca
Baldassarre
(CNR)
Vito
Trianni
(CNR)
Kyuhwa
Lee
(EPFL)