Image semantic coding using OTB Marie Liénou; TELECOM ParisTech Marine Campedel; TELECOM ParisTech

1. Competence Centre on Information Extraction
and Image Understanding for Earth Observation
July 2009
using OTB
Télécom ParisTech
Marie Liénou - Marine Campedel
Image Semantic Coding
1

2. and Image Understanding for Earth Observation
SUMMARY
Competence Centre on Information Extraction
COC
Notion of
semantic Coding
A promising
approach
Semantic Coding
OTB tool
Development of
Conclusion and an OTB tool
perspectives
2

3. and Image Understanding for Earth Observation
COC = COmpetence Centre…
Competence Centre on Information Extraction
 Tripartite agreement between CNES – DLR and Télécom
ParisTech
 Signed in June 2005
 Goal : joint action on image understanding
 SAR/Optical, HR and VHR, temporal series
 Feature extraction, modeling, indexing, compression,
(interactive) classification, interpretation, knowledge
representation, reasoning, …
 Means
 ~ 4 new phds / year
 ~10 permanent researchers partially involved
 financial support for specific actions (studentships, engineers,
post-docs)
3

4. and Image Understanding for Earth Observation
Image Semantic coding
Competence Centre on Information Extraction
Semantic Coding
Meaning Compression
Understanding Reduce data while
Interpretation ensuring informational
Image to text? content
[Barnard et al., 2003 ; Jeon et al., 2003]
[Li et Bretschneider, 2006]
Goal: find an image representation able to
capture the contained semantics
Idea: use text indexing approach + active learning
4

5. and Image Understanding for Earth Observation
Image Semantic coding
Visual interaction
Competence Centre on Information Extraction
Manual annotation
Feature
extraction
Where is
Quantization semantics?
Automatic
annotation
« visual words »
Active learning
Indexing Mining
5

6. and Image Understanding for Earth Observation
Image Semantic coding vs KIM
Competence Centre on Information Extraction
« Design and evaluation of HMC for Image Information Mining »
Daschiel and Datcu
IEEE transaction on multimedia, vol 7, no6, dec. 2005
6

7. and Image Understanding for Earth Observation
A promising approach
Competence Centre on Information Extraction
 Feature extraction
 Segmentation, arbitrar regions
 “Classical” signature: color, texture, shape descriptors
 Experiments: intensity mean and variance in each spectral band
 Quantization
 K-Means: each estimated cluster corresponds to one “visual word”
 K estimated using MDL (Minimum Description Length) descriptor
 Bag-of-words signature for semantics identification
 Count visual words on image regions which will be annotated
 Normalize (tf-idf)
 Exploitation using machine learning (SVM, LDA)
7

8. and Image Understanding for Earth Observation
A promising approach
Competence Centre on Information Extraction
 Marie Lienou PhD work (march 2009)
 Tested on several VHR (multispectral) images
 Compared to other classification approachs (GMM, SVM)
Visual word
production
Feature Classification
Quantization Count words
extraction SVM, LDA
Annotations
Feature Classification
Majority rule
extraction GMM, SVM
Low level
annotations
 Recognition accuracy demonstrated for “semantically complex”
classes Ex: “urban area”
 LDA = fast + does not need negative examples
8

9. and Image Understanding for Earth Observation
OTB tool: cocSemanticCoding
Competence Centre on Information Extraction
 Feature extraction
 Vectorial image with as many components as feature dimension
 Exploitation of OTB extractors at each pixel
 Quantization
 Use of K-Means filter
 Bag-of-words signature
 Count visual words on image regions which will be annotated
 Normalization (tf-idf)
 Learning from manual annotation
 Fluid interface facilities
 Learn LDA from only target samples
 Learn SVM from target samples and counter examples
 Classify the whole image
 Iterate
9

10. Competence Centre on Information Extraction
and Image Understanding for Earth Observation
OTB tool: cocSemanticCoding
10

11. Competence Centre on Information Extraction
and Image Understanding for Earth Observation
OTB tool: cocSemanticCoding
11

12. Competence Centre on Information Extraction
and Image Understanding for Earth Observation
OTB tool: cocSemanticCoding
12

13. Competence Centre on Information Extraction
and Image Understanding for Earth Observation
13

14. Competence Centre on Information Extraction
and Image Understanding for Earth Observation
14

15. Competence Centre on Information Extraction
and Image Understanding for Earth Observation
15

16. and Image Understanding for Earth Observation
Competence Centre on Information Extraction
Learning and classification tools
LDA on occurrence data
SVM on TFiDF data (features)
Both results can be obtained with same labeling for comparison
Difficulty for the user : compute features adapted to the underlying
semantics
16

17. and Image Understanding for Earth Observation
Conclusion
Competence Centre on Information Extraction
 OTB useful features
 Vectorial image representation
 Great diversity of available filters (extractors, classifiers)
 New = LDA classifier + estimator
 Visualization tools
 cocSemanticCoding tool availability
 www.tsi.enst.fr/~campedel/
 will be updated
 Necessity to valorize research results
 Engineering process (C++ programming)
 Not easy but OTB is a nice initiative to help researchers
 In the future: centralize processing tools (in OTB) + easy their
exploitation (documentations, interfaces)
17

18. and Image Understanding for Earth Observation
Perspectives
Competence Centre on Information Extraction
 Other COC tools should be integrated in
cocSemanticCoding
 MDL to estimate of visual words number
 new feature extractors (QMF-based texture descriptors)
 Feature selection
 Complete relevance feedback framework
 New approaches for image interpretation
 From semantics to knowledge?
 Knowledge engineering: modeling (ontologies) + reasoning
 Several works on characterizing relations between identified
concepts and/or image objects
18