SlideShare ist ein Scribd-Unternehmen logo
1 von 39
Downloaden Sie, um offline zu lesen
Hilbert problems and
                               challenges

                                       Pietro Perona
                             California Institute of Technology

                             NSF Frontiers in Vision Workshop
                              Cambridge - 21-23 Aug. 2011



Friday, August 26, 2011
Scene understanding




Friday, August 26, 2011
Scene understanding
                                   1
                                x= X
                           X       Z


                                                O
                                       x




Friday, August 26, 2011
Vision in control loop




Friday, August 26, 2011
Other sensors...




Friday, August 26, 2011
scene
                                                   vision in the loop
                                 understanding


ease of experimentation


                          goal    representation     decision/action


         active perception


             other sensors


Friday, August 26, 2011
Visual recognition



Friday, August 26, 2011
Recognizing..


                             Materials
                                          Objects




           Actions                        Scenes
Friday, August 26, 2011
Recognizing..


                             Materials
                                          Objects




           Actions                        Scenes
Friday, August 26, 2011
Recognizing..


                             Materials
                                          Objects




           Actions                        Scenes
Friday, August 26, 2011
Recognizing..


                             Materials
                                          Objects




           Actions                        Scenes
Friday, August 26, 2011
Recognizing..


                             Materials
                                          Objects




           Actions                        Scenes
Friday, August 26, 2011
Recognizing..


                             Materials
                                          Objects




           Actions                        Scenes
Friday, August 26, 2011
Recognizing..


                             Materials
                                          Objects




           Actions                        Scenes
Friday, August 26, 2011
Recognizing..


                             Materials
                                          Objects




           Actions                        Scenes
Friday, August 26, 2011
Recognizing..


                             Materials
                                          Objects




           Actions                        Scenes
Friday, August 26, 2011
Geometry and materials




Friday, August 26, 2011
How many?




Friday, August 26, 2011
Weakly
   supervised
    learning


        Transfer
        learning




Friday, August 26, 2011
Incremental
   learning




                          ...
Friday, August 26, 2011
Subordinate categorization




Friday, August 26, 2011
Organizing visual knowledge




Friday, August 26, 2011
Intentions, causes, consequences, ...




Friday, August 26, 2011
Behavior, events



Friday, August 26, 2011
Friday, August 26, 2011
Friday, August 26, 2011
Perception




                                                        PSYCHOLOGY
                          interaction, cooperation,
                                competition



                            plans, goals, behavior,
                               relationships ...


                          pose, movemes, actions,
                          activities, objects, scenes




                                                        SENSORY
                            images, trajectories




                                      World
Friday, August 26, 2011
Action                                 Perception




                                                                                                     PSYCHOLOGY
                                                                     interaction, cooperation,
 PLANNING
                          group-level goals and plans
                                                                           competition
                      SOCIAL NETWORK                                         THEORY OF SOCIOLOGY
                                                        INDIVIDUAL


                                                                       plans, goals, behavior,
                          individual goals and plans
                                                                          relationships ...
                      PREFRONTAL CORTEX                                     THEORY OF PSYCHOLOGY



                                                                     pose, movemes, actions,
 MOTOR




                               motor programs
                                                                     activities, objects, scenes




                                                                                                     SENSORY
                     MOTOR CORTEX                                                      RECOGNITION




                            sensor-based control                       images, trajectories
                     SPINAL CORD                                                 IMAGING,TRACKING




                                                                                 World
Friday, August 26, 2011
Behavior

                     • Hiearchical representation
                     • Interactions
                     • Beyond description: intentions, plans,
                          consequences,




Friday, August 26, 2011
Sharing visual
                          representations with
                                humans


Friday, August 26, 2011
Friday, August 26, 2011
Friday, August 26, 2011
Grand challenges



Friday, August 26, 2011
Drosophila behavior




                                      [Dankert et al., Nature Methods, April 2009]

Friday, August 26, 2011
Drosophila behavior




                                      [Dankert et al., Nature Methods, April 2009]

Friday, August 26, 2011
VISIPEDIA                       Users

                            Images, segments
                           annotations, links,
                            GUIs, diagnostics

                                                            Experts



Image databases
                          Annotators             Automata     Vision
                                                             scientists



Friday, August 26, 2011
Autonomous driving


                          <<show movie of traffic in India>>




Friday, August 26, 2011
Grand challenges
                     • Recognition -> Visipedia
                     • Behavior -> Fly behavior (mouse, ...human)
                     • Scene understanding -> ???
                     • Vision for action -> Autonomous driving
                          (how about manipulation?)
                     • Sustainable economy -> ???
Friday, August 26, 2011
Summary
                     • Scene understanding vs vision-for-action
                     • Recognition: just started, much to be done
                     • Behavior: ditto
                     • Sharing visual knowledge with humans??

                     • Grand challenges: many, fun and worthwhile
Friday, August 26, 2011

Weitere ähnliche Inhalte

Andere mochten auch

NIPS2007: learning using many examples
NIPS2007: learning using many examplesNIPS2007: learning using many examples
NIPS2007: learning using many examples
zukun
 
Mit6870 orsu lecture11
Mit6870 orsu lecture11Mit6870 orsu lecture11
Mit6870 orsu lecture11
zukun
 
Principal component analysis and matrix factorizations for learning (part 3) ...
Principal component analysis and matrix factorizations for learning (part 3) ...Principal component analysis and matrix factorizations for learning (part 3) ...
Principal component analysis and matrix factorizations for learning (part 3) ...
zukun
 
CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applicat...
CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applicat...CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applicat...
CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applicat...
zukun
 
Principal component analysis and matrix factorizations for learning (part 2) ...
Principal component analysis and matrix factorizations for learning (part 2) ...Principal component analysis and matrix factorizations for learning (part 2) ...
Principal component analysis and matrix factorizations for learning (part 2) ...
zukun
 
A general survey of previous works on action recognition
A general survey of previous works on action recognitionA general survey of previous works on action recognition
A general survey of previous works on action recognition
zukun
 
ECCV2010: distance function and metric learning part 2
ECCV2010: distance function and metric learning part 2ECCV2010: distance function and metric learning part 2
ECCV2010: distance function and metric learning part 2
zukun
 
15 cv mil_models_for_transformations
15 cv mil_models_for_transformations15 cv mil_models_for_transformations
15 cv mil_models_for_transformations
zukun
 
Cvpr2010 open source vision software, intro and training part vii point cloud...
Cvpr2010 open source vision software, intro and training part vii point cloud...Cvpr2010 open source vision software, intro and training part vii point cloud...
Cvpr2010 open source vision software, intro and training part vii point cloud...
zukun
 
Catalogueprofessionnel2011
Catalogueprofessionnel2011Catalogueprofessionnel2011
Catalogueprofessionnel2011
thiberge
 
Power%20 point[1]
Power%20 point[1]Power%20 point[1]
Power%20 point[1]
thiberge
 
CVPR2010: Sparse Coding and Dictionary Learning for Image Analysis: Part 3: O...
CVPR2010: Sparse Coding and Dictionary Learning for Image Analysis: Part 3: O...CVPR2010: Sparse Coding and Dictionary Learning for Image Analysis: Part 3: O...
CVPR2010: Sparse Coding and Dictionary Learning for Image Analysis: Part 3: O...
zukun
 
Fcv rep todorovic
Fcv rep todorovicFcv rep todorovic
Fcv rep todorovic
zukun
 
ECCV2008: MAP Estimation Algorithms in Computer Vision - Part 2
ECCV2008: MAP Estimation Algorithms in Computer Vision - Part 2ECCV2008: MAP Estimation Algorithms in Computer Vision - Part 2
ECCV2008: MAP Estimation Algorithms in Computer Vision - Part 2
zukun
 
Scientific Programming in Python
Scientific Programming in PythonScientific Programming in Python
Scientific Programming in Python
zukun
 

Andere mochten auch (15)

NIPS2007: learning using many examples
NIPS2007: learning using many examplesNIPS2007: learning using many examples
NIPS2007: learning using many examples
 
Mit6870 orsu lecture11
Mit6870 orsu lecture11Mit6870 orsu lecture11
Mit6870 orsu lecture11
 
Principal component analysis and matrix factorizations for learning (part 3) ...
Principal component analysis and matrix factorizations for learning (part 3) ...Principal component analysis and matrix factorizations for learning (part 3) ...
Principal component analysis and matrix factorizations for learning (part 3) ...
 
CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applicat...
CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applicat...CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applicat...
CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applicat...
 
Principal component analysis and matrix factorizations for learning (part 2) ...
Principal component analysis and matrix factorizations for learning (part 2) ...Principal component analysis and matrix factorizations for learning (part 2) ...
Principal component analysis and matrix factorizations for learning (part 2) ...
 
A general survey of previous works on action recognition
A general survey of previous works on action recognitionA general survey of previous works on action recognition
A general survey of previous works on action recognition
 
ECCV2010: distance function and metric learning part 2
ECCV2010: distance function and metric learning part 2ECCV2010: distance function and metric learning part 2
ECCV2010: distance function and metric learning part 2
 
15 cv mil_models_for_transformations
15 cv mil_models_for_transformations15 cv mil_models_for_transformations
15 cv mil_models_for_transformations
 
Cvpr2010 open source vision software, intro and training part vii point cloud...
Cvpr2010 open source vision software, intro and training part vii point cloud...Cvpr2010 open source vision software, intro and training part vii point cloud...
Cvpr2010 open source vision software, intro and training part vii point cloud...
 
Catalogueprofessionnel2011
Catalogueprofessionnel2011Catalogueprofessionnel2011
Catalogueprofessionnel2011
 
Power%20 point[1]
Power%20 point[1]Power%20 point[1]
Power%20 point[1]
 
CVPR2010: Sparse Coding and Dictionary Learning for Image Analysis: Part 3: O...
CVPR2010: Sparse Coding and Dictionary Learning for Image Analysis: Part 3: O...CVPR2010: Sparse Coding and Dictionary Learning for Image Analysis: Part 3: O...
CVPR2010: Sparse Coding and Dictionary Learning for Image Analysis: Part 3: O...
 
Fcv rep todorovic
Fcv rep todorovicFcv rep todorovic
Fcv rep todorovic
 
ECCV2008: MAP Estimation Algorithms in Computer Vision - Part 2
ECCV2008: MAP Estimation Algorithms in Computer Vision - Part 2ECCV2008: MAP Estimation Algorithms in Computer Vision - Part 2
ECCV2008: MAP Estimation Algorithms in Computer Vision - Part 2
 
Scientific Programming in Python
Scientific Programming in PythonScientific Programming in Python
Scientific Programming in Python
 

Mehr von zukun

My lyn tutorial 2009
My lyn tutorial 2009My lyn tutorial 2009
My lyn tutorial 2009
zukun
 
ETHZ CV2012: Tutorial openCV
ETHZ CV2012: Tutorial openCVETHZ CV2012: Tutorial openCV
ETHZ CV2012: Tutorial openCV
zukun
 
ETHZ CV2012: Information
ETHZ CV2012: InformationETHZ CV2012: Information
ETHZ CV2012: Information
zukun
 
Siwei lyu: natural image statistics
Siwei lyu: natural image statisticsSiwei lyu: natural image statistics
Siwei lyu: natural image statistics
zukun
 
Lecture9 camera calibration
Lecture9 camera calibrationLecture9 camera calibration
Lecture9 camera calibration
zukun
 
Brunelli 2008: template matching techniques in computer vision
Brunelli 2008: template matching techniques in computer visionBrunelli 2008: template matching techniques in computer vision
Brunelli 2008: template matching techniques in computer vision
zukun
 
Modern features-part-4-evaluation
Modern features-part-4-evaluationModern features-part-4-evaluation
Modern features-part-4-evaluation
zukun
 
Modern features-part-3-software
Modern features-part-3-softwareModern features-part-3-software
Modern features-part-3-software
zukun
 
Modern features-part-2-descriptors
Modern features-part-2-descriptorsModern features-part-2-descriptors
Modern features-part-2-descriptors
zukun
 
Modern features-part-1-detectors
Modern features-part-1-detectorsModern features-part-1-detectors
Modern features-part-1-detectors
zukun
 
Modern features-part-0-intro
Modern features-part-0-introModern features-part-0-intro
Modern features-part-0-intro
zukun
 
Lecture 02 internet video search
Lecture 02 internet video searchLecture 02 internet video search
Lecture 02 internet video search
zukun
 
Lecture 01 internet video search
Lecture 01 internet video searchLecture 01 internet video search
Lecture 01 internet video search
zukun
 
Lecture 03 internet video search
Lecture 03 internet video searchLecture 03 internet video search
Lecture 03 internet video search
zukun
 
Icml2012 tutorial representation_learning
Icml2012 tutorial representation_learningIcml2012 tutorial representation_learning
Icml2012 tutorial representation_learning
zukun
 
Advances in discrete energy minimisation for computer vision
Advances in discrete energy minimisation for computer visionAdvances in discrete energy minimisation for computer vision
Advances in discrete energy minimisation for computer vision
zukun
 
Gephi tutorial: quick start
Gephi tutorial: quick startGephi tutorial: quick start
Gephi tutorial: quick start
zukun
 
EM algorithm and its application in probabilistic latent semantic analysis
EM algorithm and its application in probabilistic latent semantic analysisEM algorithm and its application in probabilistic latent semantic analysis
EM algorithm and its application in probabilistic latent semantic analysis
zukun
 
Object recognition with pictorial structures
Object recognition with pictorial structuresObject recognition with pictorial structures
Object recognition with pictorial structures
zukun
 
Iccv2011 learning spatiotemporal graphs of human activities
Iccv2011 learning spatiotemporal graphs of human activities Iccv2011 learning spatiotemporal graphs of human activities
Iccv2011 learning spatiotemporal graphs of human activities
zukun
 

Mehr von zukun (20)

My lyn tutorial 2009
My lyn tutorial 2009My lyn tutorial 2009
My lyn tutorial 2009
 
ETHZ CV2012: Tutorial openCV
ETHZ CV2012: Tutorial openCVETHZ CV2012: Tutorial openCV
ETHZ CV2012: Tutorial openCV
 
ETHZ CV2012: Information
ETHZ CV2012: InformationETHZ CV2012: Information
ETHZ CV2012: Information
 
Siwei lyu: natural image statistics
Siwei lyu: natural image statisticsSiwei lyu: natural image statistics
Siwei lyu: natural image statistics
 
Lecture9 camera calibration
Lecture9 camera calibrationLecture9 camera calibration
Lecture9 camera calibration
 
Brunelli 2008: template matching techniques in computer vision
Brunelli 2008: template matching techniques in computer visionBrunelli 2008: template matching techniques in computer vision
Brunelli 2008: template matching techniques in computer vision
 
Modern features-part-4-evaluation
Modern features-part-4-evaluationModern features-part-4-evaluation
Modern features-part-4-evaluation
 
Modern features-part-3-software
Modern features-part-3-softwareModern features-part-3-software
Modern features-part-3-software
 
Modern features-part-2-descriptors
Modern features-part-2-descriptorsModern features-part-2-descriptors
Modern features-part-2-descriptors
 
Modern features-part-1-detectors
Modern features-part-1-detectorsModern features-part-1-detectors
Modern features-part-1-detectors
 
Modern features-part-0-intro
Modern features-part-0-introModern features-part-0-intro
Modern features-part-0-intro
 
Lecture 02 internet video search
Lecture 02 internet video searchLecture 02 internet video search
Lecture 02 internet video search
 
Lecture 01 internet video search
Lecture 01 internet video searchLecture 01 internet video search
Lecture 01 internet video search
 
Lecture 03 internet video search
Lecture 03 internet video searchLecture 03 internet video search
Lecture 03 internet video search
 
Icml2012 tutorial representation_learning
Icml2012 tutorial representation_learningIcml2012 tutorial representation_learning
Icml2012 tutorial representation_learning
 
Advances in discrete energy minimisation for computer vision
Advances in discrete energy minimisation for computer visionAdvances in discrete energy minimisation for computer vision
Advances in discrete energy minimisation for computer vision
 
Gephi tutorial: quick start
Gephi tutorial: quick startGephi tutorial: quick start
Gephi tutorial: quick start
 
EM algorithm and its application in probabilistic latent semantic analysis
EM algorithm and its application in probabilistic latent semantic analysisEM algorithm and its application in probabilistic latent semantic analysis
EM algorithm and its application in probabilistic latent semantic analysis
 
Object recognition with pictorial structures
Object recognition with pictorial structuresObject recognition with pictorial structures
Object recognition with pictorial structures
 
Iccv2011 learning spatiotemporal graphs of human activities
Iccv2011 learning spatiotemporal graphs of human activities Iccv2011 learning spatiotemporal graphs of human activities
Iccv2011 learning spatiotemporal graphs of human activities
 

Kürzlich hochgeladen

Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
FIDO Alliance
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
panagenda
 

Kürzlich hochgeladen (20)

AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream Processing
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
Using IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandUsing IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & Ireland
 

Fcv taxo perona