For the full video of this presentation, please visit:
http://www.embedded-vision.com/platinum-members/embedded-vision-alliance/embedded-vision-training/videos/pages/dec-2016-member-meeting-uofw
For more information about embedded vision, please visit:
http://www.embedded-vision.com
Professor Jeff Bilmes of the University of Washington delivers the presentation "Image and Video Summarization" at the December 2016 Embedded Vision Alliance Member Meeting. Bilmes provides an overview of the state of the art in image and video summarization.
Elevate Developer Efficiency & build GenAI Application with Amazon Q
"Image and Video Summarization," a Presentation from the University of Washington
1. Bigness Info. in Data Image Summarization Video Summarization End
Image and Video Summarization
Jeffrey A. Bilmes
Professor
Departments of Electrical Engineering
& Computer Science and Engineering
University of Washington, Seattle
http://melodi.ee.washington.edu/~bilmes
and
Visiting Scientist, Google Research
Wednesday, Dec 7th, 2016
J. Bilmes Image and Video Summarization — 12/7/2016 page 1 / 35
2. Bigness Info. in Data Image Summarization Video Summarization End
Outline
1 Bigness
2 Information In Data
3 Image Summarization
4 Video Summarization
5 End
J. Bilmes Image and Video Summarization — 12/7/2016 page 2 / 35
3. Bigness Info. in Data Image Summarization Video Summarization End
Summarization
What is summarization?
Why do we need summarization?
J. Bilmes Image and Video Summarization — 12/7/2016 page 4 / 35
4. Bigness Info. in Data Image Summarization Video Summarization End
Bigness
J. Bilmes Image and Video Summarization — 12/7/2016 page 5 / 35
5. Bigness Info. in Data Image Summarization Video Summarization End
Water
H2O molecules
small (n-body)
medium (fluid dynamics, viscosity, compressibility),
large (global weather systems, meteorology).
Same underlying molecular collision events!
J. Bilmes Image and Video Summarization — 12/7/2016 page 6 / 35
6. Bigness Info. in Data Image Summarization Video Summarization End
Neurons
Neurons
small (neural spike trains, population coding)
medium (intelligence, consciousness, psychology)
large (society, social choice, wisdom of the crowd)
Same underlying electrical and chemical impulses.
J. Bilmes Image and Video Summarization — 12/7/2016 page 7 / 35
7. Bigness Info. in Data Image Summarization Video Summarization End
Bigger is Different
“More is Different”, P.W. Anderson, 1972
(Nobel laureate). “The ability to reduce
everything to simple fundamental laws does
not imply the ability to start from those laws
and reconstruct the universe.”
“. . . alterations of being . . . are not only the
transition of one magnitude into another,
but a transition from quantity into quality,”
Hegel, The Science of Logic, 1816
J. Bilmes Image and Video Summarization — 12/7/2016 page 8 / 35
8. Bigness Info. in Data Image Summarization Video Summarization End
Big Data is Different Data
Hypothesis: extremely large data sets offer qualitatively different
capabilities than small data sets.
Evidence: Image Completion (Hays & Efros, 2007)
“our initial experiments . . . on a dataset of ten thousand images
were very discouraging. However, increasing the image collection
to two million yielded a qualitative leap in performance”
J. Bilmes Image and Video Summarization — 12/7/2016 page 9 / 35
9. Bigness Info. in Data Image Summarization Video Summarization End
Modern Times
Big Data is Big and Getting Even Bigger.
Sensors &
Devices
Social Media
VoIP
Enterprise
Data
Now A Few Years
From Now
A Few Years
Ago
VolumeofData
2.5 quintillion bytes (2.5 million terabytes) of data/day (IBM)
>90% of the world’s data has been created in the last two years.
J. Bilmes Image and Video Summarization — 12/7/2016 page 10 / 35
10. Bigness Info. in Data Image Summarization Video Summarization End
Big Data in Machine Learning
Statistics, Machine Learning, and Artificial Intelligence (AI)
“There’s no data like more data”,
Computational Consequences:
Massive computational resource demands!
Research opportunities to address new computational challenges
1 systems programming, parallel and distributed computing,
network topologies, efficient databases, GPUs.
2 Examples: map reduce, Hadoop, GraphLab, HaLoop, Greenplum,
Asterix, Spark, SystemML, MLBase, Myria, etc.
J. Bilmes Image and Video Summarization — 12/7/2016 page 11 / 35
11. Bigness Info. in Data Image Summarization Video Summarization End
The Two Foundations of Big Data?
Data
StatisticalSignificance
ParallelComputingSystems
??????????????
Large Data
StatisticalSignificance
Larg
J. Bilmes Image and Video Summarization — 12/7/2016 page 12 / 35
12. Bigness Info. in Data Image Summarization Video Summarization End
Goal: Data to Information
Data is:
Streaming
Torrential
Relentless
Multi-modal
Mostly Unstructured
Sensors/Actuators
Redundant
High Dimensional
Distributed
J. Bilmes Image and Video Summarization — 12/7/2016 page 13 / 35
13. Bigness Info. in Data Image Summarization Video Summarization End
Big Data is Different Data: A Proposition
Hypothesis: extremely large data sets offer
qualitatively different capabilities than small
data sets.
Problem: Big data sets are big, unwieldy,
computationally challenging, and often highly
redundant.
Research Quest: Can statistical predictions and
actions be made cost effectively using the right
data management strategies?
Yes, by reducing redundancy.
J. Bilmes Image and Video Summarization — 12/7/2016 page 14 / 35
14. Bigness Info. in Data Image Summarization Video Summarization End
How to identify and measure redundancy?
J. Bilmes Image and Video Summarization — 12/7/2016 page 15 / 35
15. Bigness Info. in Data Image Summarization Video Summarization End
How to identify and measure redundancy?
J. Bilmes Image and Video Summarization — 12/7/2016 page 16 / 35
16. Bigness Info. in Data Image Summarization Video Summarization End
Measuring Information in Data
What is information?
Information Theory (entropy, mutual information) ⇔ probability
distributions.
Kolmogorov Complexity ⇔ algorithms & models of computation.
Information measures over non-random data samples (e.g., images).
f( ) = f( , )
< f( , ) < f( , )
J. Bilmes Image and Video Summarization — 12/7/2016 page 18 / 35
17. Bigness Info. in Data Image Summarization Video Summarization End
Measuring Information in Data
Diminishing returns:
The more you have, the less valuable is anything you don’t have.
f( )f( )-
f( )
- f( )
≥
J. Bilmes Image and Video Summarization — 12/7/2016 page 19 / 35
18. Bigness Info. in Data Image Summarization Video Summarization End
Example: Number of Colors of Balls in Urns
Consider an urn containing colored balls. Given a set S of balls,
f (S) counts the number of distinct colors.
Initial value: 2 (colors in urn).
New value with added blue ball: 3
Initial value: 3 (colors in urn).
New value with added blue ball: 3
Submodularity : Incremental Value of Object Diminishes in a
Larger Context (diminishing returns). Thus, f is submodular.
J. Bilmes Image and Video Summarization — 12/7/2016 page 20 / 35
19. Bigness Info. in Data Image Summarization Video Summarization End
As the data set size grow . . .
There is no data like more data ⇒ more data is like no more data.
From Andrew Ng’s Stanford
machine learning class, 2011
J. Bilmes Image and Video Summarization — 12/7/2016 page 21 / 35
20. Bigness Info. in Data Image Summarization Video Summarization End
As the data set sizes grow . . .
0.70
0.75
0.80
0.85
0.90
0.95
1.00
0.1 1 10 100 1000
Millions of Words
TestAccuracy
Memory-Based
Winnow
Perceptron
Naïve Bayes
Banko & Brill 2001
(Riccardi & Hakkani-Tür, 2005, Speech Recognition) (Callison-Burch&Bloodgood, 2010, Machine Translation)
Tong & Koller, 2001 (Soon, Ng, Lim, 2001, Coreference Resolution)
50 100 150 200 250
Number of Training Examples
0
0.2
0.4
0.6
0.8
1
Accuracy
(Kadie, 1995, Generic Classification)
Sentiment Tutorial, Chris Potts, Stanford Ling., 2011
78
80
82
84
86
88
90
92
700 1.750 3.500 7.000 10.500 14.000 17.500 21.000
Overallaccuracy(%)
Training set size
SVMs
DTs
( Kavzoglu & Colkesen, 2012, Image Classification)
Speed, memory, attention, problem solving
playing game Luminosity
http://www.lumosity.com/blog/how-much-and-how-often-should-i-train/
J. Bilmes Image and Video Summarization — 12/7/2016 page 22 / 35
21. Bigness Info. in Data Image Summarization Video Summarization End
Submodularity and Learning Curves
Proposition
Let V = {1, 2, . . . , n} be a finite ground set, and let f : 2V
→ R be a
set function. If for all permutations σ of V , we have that for all i ≤ j:
f (σj |Si−1) ≥ f (σj |Sj−1) (1)
with Si = {σ1, σ2, . . . , σi }, then f is submodular.
Learning curves might not be exactly submodular, but
submodularity seems a reasonable model.
J. Bilmes Image and Video Summarization — 12/7/2016 page 23 / 35
22. Bigness Info. in Data Image Summarization Video Summarization End
What is Summarization?
1 Start with a massive data set (images, videos, etc.), set V .
2 Identify a good (submodular) information function f (by hand or
by machine learning) that represents information in V .
3 Find a subset A ⊆ V of a given size that retains as much
information as possible.
4 Luckily, this normally exponential time problem can be done
computationally efficiently!!
J. Bilmes Image and Video Summarization — 12/7/2016 page 24 / 35
23. Bigness Info. in Data Image Summarization Video Summarization End
Modern Image Collections
Many images, also that have a higher level gestalt than just a few.
J. Bilmes Image and Video Summarization — 12/7/2016 page 26 / 35
24. Bigness Info. in Data Image Summarization Video Summarization End
Image Summarization
Task: Summarize collection
of images by representative
subset of the images
Applications:
Summarizing your
holiday pictures.
Summarizing image
search results
Efficient browsing of
image collections
Video frame
summarization
Difficulties:
No high level
⇓
J. Bilmes Image and Video Summarization — 12/7/2016 page 27 / 35
25. Bigness Info. in Data Image Summarization Video Summarization End
Image Summarization - Data Collection
Data Statistics
14 image collections with 100 pictures each
∼ 400 human summaries for every image collection, via Amazon
Turk, about 5500 summaries total!
Example collections:
J. Bilmes Image and Video Summarization — 12/7/2016 page 28 / 35
26. Bigness Info. in Data Image Summarization Video Summarization End
Image Summarization
Whole collection: 3 best summaries:
3 medium summaries:
3 worst summaries:
J. Bilmes Image and Video Summarization — 12/7/2016 page 29 / 35
27. Bigness Info. in Data Image Summarization Video Summarization End
Image Summarization
Typical Results - Learnt mixture using Max-Margin
f(∅) = 0 f(V ) = 1
Greedy Min
Average Pruned Random
Max of Learned Mixture
Average Pruned Human
Greedy Max
Average Pruned Random
Average Pruned Human
J. Bilmes Image and Video Summarization — 12/7/2016 page 30 / 35
28. Bigness Info. in Data Image Summarization Video Summarization End
Real-Time Running Online Video Summarization
Live Video
Feed
Most recent
representative video
snippet (repeating)
Next most recent
representative video
snippet (repeating)
Third most recent
representative video
snippet (repeating)
Fourth most recent
representative video
snippet (repeating)
Least recent
representative video
snippet (repeating)
...
J. Bilmes Image and Video Summarization — 12/7/2016 page 32 / 35
29. Bigness Info. in Data Image Summarization Video Summarization End
Real-Time Running Online Video Summarization
J. Bilmes Image and Video Summarization — 12/7/2016 page 33 / 35
30. Bigness Info. in Data Image Summarization Video Summarization End
The End
The End: Thank you!
++
+ +
f(A) f(B) f(A ∪ B)
= f(Ar ) +f(C) + f(Br )
≥
≥
= f(A ∩ B)
f(A ∩ B)
= f(Ar ) + 2f(C) + f(Br )
J. Bilmes Image and Video Summarization — 12/7/2016 page 35 / 35