1. A Large Scale Concept Ontology
for Multimedia Understanding
Milind Naphade, John R. Smith, Alexander Hauptmann, Shih-Fu Chang &
Edward Chang
IBM Research, Carnegie Mellon University, Columbia University & University of California at Santa Barbara
naphade@us.ibm.com jsmith@us.ibm.com alex@cs.cmu.edu sfchang@ee.columbia.edu
echang@xanadu.ece.ucsb.edu
April 2005
NRRC NWRRC
MITRE
2. Central Idea
• Collaborative activity of three
critical communities – Users,
Library Scientists and Knowledge
Experts, and Technical
Researchers, Algorithm, System
and Solution Designers – to create
a user-driven concept ontology for
analysis of video broadcast news
Page
PNN MITRE
3. Central Idea
• Collaborative activity of three Users (Analysts,
critical communities – Users, Broadcasters)
Intelligence
Library Scientists and Knowledge
Community,
Experts, and Technical
Broadcasting
Researchers, Algorithm, System
Corporations
and Solution Designers – to create
a user-driven concept ontology for
analysis of video broadcast news
Page
PNN MITRE
4. Central Idea
• Collaborative activity of three Users (Analysts,
critical communities – Users, Broadcasters)
Intelligence
Library Scientists and Knowledge
Community,
Experts, and Technical
Broadcasting
Researchers, Algorithm, System
Corporations
and Solution Designers – to create
a user-driven concept ontology for
analysis of video broadcast news
Vision, Machine
Learning, Detection
Analytics
Technical Researchers, Algorithm
Designers & System Developers
Page
PNN MITRE
5. Central Idea
• Collaborative activity of three Users (Analysts,
critical communities – Users, Broadcasters)
Intelligence
Library Scientists and Knowledge
Community,
Experts, and Technical
Broadcasting
Researchers, Algorithm, System
Corporations
and Solution Designers – to create
a user-driven concept ontology for
analysis of video broadcast news
Knowledge
Vision, Machine Representation,
Learning, Detection Library Scientists
Analytics Standardization
Technical Researchers, Algorithm Ontology Experts
Designers & System Developers
Page
PNN MITRE
6. Central Idea
• Collaborative activity of three Users (Analysts,
critical communities – Users, Broadcasters)
Intelligence
Library Scientists and Knowledge
Community,
Experts, and Technical
Broadcasting
Researchers, Algorithm, System
Corporations
and Solution Designers – to create
a user-driven concept ontology for
analysis of video broadcast news
Lexicon and
Ontology
1000 or more
Knowledge
concepts
Vision, Machine Representation,
Learning, Detection Library Scientists
Analytics Standardization
Technical Researchers, Algorithm Ontology Experts
Designers & System Developers
Page
PNN MITRE
7. Problem
• Users and analysts require richly annotated video content for
accomplishing required access and analysis functions over
massive amount of video content.
• Big Barriers:
- Research community needs to advance technology for
bridging gap from low-level features to semantics
- Lack of large scale useful well-defined semantic lexicon
- Lack of user-centric ontology
- Lack of corpora annotated with rich lexicon
- Lack of feasibility studies for any ontology if defined
• Examples:
- The TRECVID lexicon defined from a frequentist
perspective. Its not user-centric.
• No effort to date to design lexicon by joint partnership between
different communities (users, knowledge experts, technical)
Page
PNN MITRE
9. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
Page
PNN MITRE
10. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
Page
PNN MITRE
11. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
Page
PNN MITRE
12. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
Page
PNN MITRE
13. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
Page
PNN MITRE
14. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
- Develop draft concept ontology for video broadcast news domain
Page
PNN MITRE
15. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
- Develop draft concept ontology for video broadcast news domain
- Solicit input on technical capabilities
Page
PNN MITRE
16. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
- Develop draft concept ontology for video broadcast news domain
- Solicit input on technical capabilities
- Analyze technical capabilities for concept modeling and detection
Page
PNN MITRE
17. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
- Develop draft concept ontology for video broadcast news domain
- Solicit input on technical capabilities
- Analyze technical capabilities for concept modeling and detection
- Form benchmark and define annotation tasks
Page
PNN MITRE
18. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
- Develop draft concept ontology for video broadcast news domain
- Solicit input on technical capabilities
- Analyze technical capabilities for concept modeling and detection
- Form benchmark and define annotation tasks
- Annotate benchmark dataset
Page
PNN MITRE
19. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
- Develop draft concept ontology for video broadcast news domain
- Solicit input on technical capabilities
- Analyze technical capabilities for concept modeling and detection
- Form benchmark and define annotation tasks
- Annotate benchmark dataset
- Perform benchmark concept modeling, detection and evaluation
Page
PNN MITRE
20. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
- Develop draft concept ontology for video broadcast news domain
- Solicit input on technical capabilities
- Analyze technical capabilities for concept modeling and detection
- Form benchmark and define annotation tasks
- Annotate benchmark dataset
- Perform benchmark concept modeling, detection and evaluation
- Analyze concept detection performance and revise concept ontology
Page
PNN MITRE
21. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
- Develop draft concept ontology for video broadcast news domain
- Solicit input on technical capabilities
- Analyze technical capabilities for concept modeling and detection
- Form benchmark and define annotation tasks
- Annotate benchmark dataset
- Perform benchmark concept modeling, detection and evaluation
- Analyze concept detection performance and revise concept ontology
- Conduct gap analysis and identify outstanding research challenges
Page
PNN MITRE
23. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
Page
PNN MITRE
24. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
Page
PNN MITRE
25. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
Page
PNN MITRE
26. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
Page
PNN MITRE
27. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
Page
PNN MITRE
28. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
Page
PNN MITRE
29. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
Page
PNN MITRE
30. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
- Ad hoc Tasks
Page
PNN MITRE
31. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
- Ad hoc Tasks
• Task 1: Annotation
Page
PNN MITRE
32. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
- Ad hoc Tasks
• Task 1: Annotation
• Task 2: Experimentation
Page
PNN MITRE
33. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
- Ad hoc Tasks
• Task 1: Annotation
• Task 2: Experimentation
• Task 3: Evaluation
Page
PNN MITRE
34. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
- Ad hoc Tasks
• Task 1: Annotation
• Task 2: Experimentation
• Task 3: Evaluation
- Ontology Evaluation Workshop (two-weeks):
Page
PNN MITRE
35. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
- Ad hoc Tasks
• Task 1: Annotation
• Task 2: Experimentation
• Task 3: Evaluation
- Ontology Evaluation Workshop (two-weeks):
• Part 1: Validation and Refinement
Page
PNN MITRE
36. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
- Ad hoc Tasks
• Task 1: Annotation
• Task 2: Experimentation
• Task 3: Evaluation
- Ontology Evaluation Workshop (two-weeks):
• Part 1: Validation and Refinement
• Part 2: Outstanding Challenges and Recommendations
Page
PNN MITRE
37. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
- Ad hoc Tasks
• Task 1: Annotation
• Task 2: Experimentation
• Task 3: Evaluation
- Ontology Evaluation Workshop (two-weeks):
• Part 1: Validation and Refinement
• Part 2: Outstanding Challenges and Recommendations
• Substantial off-line tasks for annotation and experimentation require
Page
PNN MITRE
38. Broadcast News Video Content Description Ontology
• Why the Focus on Broadcast News Domain?
Broadcast News Ontology
- Critical mass of users, content providers, applications
- Good content availability (TRECVID, LDC, FBIS)
- Shares large set of core concepts with other domains
•
News
Ontology Formalism:
Production Broadcast
Grammars
- Entity-Relationship (E-R) Graphs
News
Content
- RDF, DAML / DAML+OIL, W3C OWL
News
- MPEG-7, MediaNet, VEML
Domain
• Seed Representations:
- TRECVID-2003 News Lexicon (Annotation Forum)
Core
- Library of Congress TGM-I
Video
- CNN, BBC Classification Systems
MPEG-7 Video Annotation Tool
Page
PNN MITRE
39. Broadcast News Video Content Description Ontology
• Why the Focus on Broadcast News Domain?
Broadcast News Ontology
- Critical mass of users, content providers, applications
- Good content availability (TRECVID, LDC, FBIS)
- Shares large set of core concepts with other domains
•
News
Ontology Formalism:
Production Broadcast
Grammars
- Entity-Relationship (E-R) Graphs
News
Content
- RDF, DAML / DAML+OIL, W3C OWL
News
- MPEG-7, MediaNet, VEML
Domain
• Seed Representations:
- TRECVID-2003 News Lexicon (Annotation Forum)
Core
- Library of Congress TGM-I
Video
- CNN, BBC Classification Systems
Concepts
Objects Sites
Actions
Person
Outdoors Indoors
People
Face
News
Monolog
News News
Anchor Studio
Subject Dialog
Crowd
MPEG-7 Video Annotation Tool
Page
PNN MITRE
40. Broadcast News Video Content Description Ontology
• Why the Focus on Broadcast News Domain?
Broadcast News Ontology
- Critical mass of users, content providers, applications
- Good content availability (TRECVID, LDC, FBIS)
- Shares large set of core concepts with other domains
•
News
Ontology Formalism:
Production Broadcast
Grammars
- Entity-Relationship (E-R) Graphs
News
Content
- RDF, DAML / DAML+OIL, W3C OWL
News
- MPEG-7, MediaNet, VEML
Domain
• Seed Representations:
- TRECVID-2003 News Lexicon (Annotation Forum)
Core
- Library of Congress TGM-I
Video
- CNN, BBC Classification Systems
Concepts
Objects Sites
Actions
Person
Outdoors Indoors
People
Face
News
Monolog
News News
Anchor Studio
Subject Dialog
Crowd
MPEG-7 Video Annotation Tool
Page
PNN MITRE
43. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
Page
PNN MITRE
44. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Page
PNN MITRE
45. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
Page
PNN MITRE
46. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
Page
PNN MITRE
47. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Page
PNN MITRE
48. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
Page
PNN MITRE
49. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
• Requirements and Existing Practices
Page
PNN MITRE
50. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
• Requirements and Existing Practices
• Domain Concepts and Ontology System
Page
PNN MITRE
51. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
• Requirements and Existing Practices
• Domain Concepts and Ontology System
• Video Concept Ontology
Page
PNN MITRE
52. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
• Requirements and Existing Practices
• Domain Concepts and Ontology System
• Video Concept Ontology
Part 2: Technical Analysis
Page
PNN MITRE
53. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
• Requirements and Existing Practices
• Domain Concepts and Ontology System
• Video Concept Ontology
Part 2: Technical Analysis
• Analyze technical capabilities for concept modeling and
detection
Page
PNN MITRE
54. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
• Requirements and Existing Practices
• Domain Concepts and Ontology System
• Video Concept Ontology
Part 2: Technical Analysis
• Analyze technical capabilities for concept modeling and
detection
• Form benchmark and define annotation tasks
Page
PNN MITRE
55. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
• Requirements and Existing Practices
• Domain Concepts and Ontology System
• Video Concept Ontology
Part 2: Technical Analysis
• Analyze technical capabilities for concept modeling and
detection
• Form benchmark and define annotation tasks
Output: Version 1
Page
PNN MITRE
56. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
• Requirements and Existing Practices
• Domain Concepts and Ontology System
• Video Concept Ontology
Part 2: Technical Analysis
• Analyze technical capabilities for concept modeling and
detection
• Form benchmark and define annotation tasks
Output: Version 1
• Benchmark (Use cases, Annotation)
Page
PNN MITRE
59. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
Page
PNN MITRE
60. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Page
PNN MITRE
61. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
Page
PNN MITRE
62. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Page
PNN MITRE
63. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
Page
PNN MITRE
64. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Page
PNN MITRE
65. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
Page
PNN MITRE
66. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
Page
PNN MITRE
67. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
Page
PNN MITRE
68. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Page
PNN MITRE
69. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
Page
PNN MITRE
70. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Page
PNN MITRE
71. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
Page
PNN MITRE
72. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
- Analyze evaluation of ontology, concept detection and its application to use case
answering.
Page
PNN MITRE
73. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
- Analyze evaluation of ontology, concept detection and its application to use case
answering.
Output
Page
PNN MITRE
74. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
- Analyze evaluation of ontology, concept detection and its application to use case
answering.
Output
- Domain Concepts v.2 and Ontology System v.2
Page
PNN MITRE
75. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
- Analyze evaluation of ontology, concept detection and its application to use case
answering.
Output
- Domain Concepts v.2 and Ontology System v.2
- Video Concept Ontology v.2
Page
PNN MITRE
76. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
- Analyze evaluation of ontology, concept detection and its application to use case
answering.
Output
- Domain Concepts v.2 and Ontology System v.2
- Video Concept Ontology v.2
Part 2: Outstanding Challenges
Page
PNN MITRE
77. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
- Analyze evaluation of ontology, concept detection and its application to use case
answering.
Output
- Domain Concepts v.2 and Ontology System v.2
- Video Concept Ontology v.2
Part 2: Outstanding Challenges
- Conduct gap analysis and identify outstanding research challenges
Page
PNN MITRE
78. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
- Analyze evaluation of ontology, concept detection and its application to use case
answering.
Output
- Domain Concepts v.2 and Ontology System v.2
- Video Concept Ontology v.2
Part 2: Outstanding Challenges
- Conduct gap analysis and identify outstanding research challenges
Output:
Page
PNN MITRE
79. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
- Analyze evaluation of ontology, concept detection and its application to use case
answering.
Output
- Domain Concepts v.2 and Ontology System v.2
- Video Concept Ontology v.2
Part 2: Outstanding Challenges
- Conduct gap analysis and identify outstanding research challenges
Output:
- Research Challenges v.1
Page
PNN MITRE
87. Ontology
Query Evaluation
Evaluation v.1
Concept with Automatic
Detection Workshop 2: Evaluation
Detection v.1
Evaluation v.1
Analysis
• Revises
• Revises lexicon ontology system
Ontology
Domain
based on based on
System
Concepts
performance performance
Study v.2
Study v.2
analysis analysis
Ontology
Re-design
Requirements
Study v.1
• Refines lexicon
Video Concept
and ontology for
Ontology
broadcast news
v.2
video domain
Input
Task
s
Output
Documents
Page
PNN MITRE
88. Ontology
Query Evaluation
Evaluation v.1
Concept with Automatic
Detection Workshop 2: Evaluation
Detection v.1
Evaluation v.1
Analysis
• Revises
• Revises lexicon ontology system
Ontology
Domain
based on based on
System
Concepts
performance performance
Study v.2
Study v.2
analysis analysis
Ontology
Re-design
Requirements Workshop 2:
Study v.1
Outstanding Challenges
• Refines lexicon
Video Concept
and ontology for
Ontology
broadcast news
v.2
video domain
Gap
Analysis
Input
Task
• Identifies and
s
• Recommendations defines
Recommendations Research technology gaps
for ontology
v.1 Challenge v.1 and challenges
exploitation and
Output
for future
solution design
Documents
research
Page
PNN MITRE
89. Domain and Data Sets
• Candidate data set:
- TRECVID Corpus (>200 hours of video broadcast news from CNN
and ABC). Has the following advantages
• availability
• generalization capability better with than other domains
• # of research groups up to speed on this domain for tools/detectors
• TREC established some benchmark and evaluation metrics already.
- Will avoid letting domain specifics influence the design of ontology to an extent where
the ontology starts catering to artifacts of the BN domain.
- Will seek other sources such as FBIS, WNC etc.
• Annotation issues:
- Plan to leverage prior video annotation efforts where possible (e.g.,
TRECVID annotation forum)
- Hands-on annotation effort will induce discussions and requires
refinements of concepts meanings
Page
PNN MITRE
90. Evaluation Methods
• Require benchmarks and metrics for evaluating:
- Utility of ontology – coverage of queries in terms of quality and quantity
- Feasibility of ontology:
• Accuracy of concept detection and degree of automation (amount of
training)
• Effectiveness of query systems using automatically extracted
concepts
• Metrics of Retrieval Effectiveness
- Precision & Recall Curves, Average Precision, Precision at Fixed Depth
• Metrics of Lexicon Effectiveness
- Number of Use Cases that can be answered by lexicon successfully
- Mean average precision across the set of use cases
• Evaluate at multiple levels of granularity:
- Individual concept, classes, hierarchies
Page
PNN MITRE
92. Confirmed Participants – Knowledge Experts and Users
Library Sciences and Knowledge
representation (definition of
lexicon):
Corrine Jorgensen, School of
Information Studies, Florida State
University
Barbara Tillett, Chief of Cataloging
Policy and Support, Library of
Congress
Jerry Hobbs, USC / ISI
Michael Witbrock, Cycorp
Ronald Murray, Preservation
Reformatting Division, Library of
Congress
Page
PNN MITRE
93. Confirmed Participants – Knowledge Experts and Users
Library Sciences and Knowledge
representation (definition of
lexicon):
Corrine Jorgensen, School of
Information Studies, Florida State
University
Barbara Tillett, Chief of Cataloging
Policy and Support, Library of
Congress
Jerry Hobbs, USC / ISI
Michael Witbrock, Cycorp
Ronald Murray, Preservation
Reformatting Division, Library of
Congress
R&D Agencies
John Prange, ARDA
Sankar Basu, Div. of Computing and
Comm. Foundations, NSF
Maria Zemankova, Div. of Inform. and
Intell. Systems., NSF
Page
PNN MITRE
94. Confirmed Participants – Knowledge Experts and Users
Library Sciences and Knowledge Standardization and Benchmarking
representation (definition of (theoretical and empirical
lexicon): evaluation):
Corrine Jorgensen, School of Paul Over, NIST
Information Studies, Florida State John Garofolo, NIST
University Donna Harman, NIST
Barbara Tillett, Chief of Cataloging David Day, MITRE
Policy and Support, Library of John R. Smith, IBM Research
Congress
Jerry Hobbs, USC / ISI
Michael Witbrock, Cycorp
Ronald Murray, Preservation
Reformatting Division, Library of
Congress
R&D Agencies
John Prange, ARDA
Sankar Basu, Div. of Computing and
Comm. Foundations, NSF
Maria Zemankova, Div. of Inform. and
Intell. Systems., NSF
Page
PNN MITRE
95. Confirmed Participants – Knowledge Experts and Users
Library Sciences and Knowledge Standardization and Benchmarking
representation (definition of (theoretical and empirical
lexicon): evaluation):
Corrine Jorgensen, School of Paul Over, NIST
Information Studies, Florida State John Garofolo, NIST
University Donna Harman, NIST
Barbara Tillett, Chief of Cataloging David Day, MITRE
Policy and Support, Library of John R. Smith, IBM Research
Congress
Jerry Hobbs, USC / ISI
Michael Witbrock, Cycorp
Ronald Murray, Preservation
Reformatting Division, Library of
User Communities (interpretation of
Congress
use cases for lexicon definition,
broadcasters help getting query logs
for finding useful lexical entries)
R&D Agencies
Joanne Evans, British Broadcasting
John Prange, ARDA
Corporation
Sankar Basu, Div. of Computing and
Chris Porter, Getty Images
Comm. Foundations, NSF
ARDA and analysts
Maria Zemankova, Div. of Inform. and
Intell. Systems., NSF
Page
PNN MITRE
96. Confirmed Participants – Technical Team
Theoretical Analysis: Experimentation: (Help Prototyping: (Help with
address evaluation issues prototyping tools for
(Help conduct analysis
for lexicon, ontology and annotation, evaluation,
during initial lexicon and
concept evaluation) querying, summarization
ontology design) Alexander and statistics gathering)
Shih-Fu Chang,
Hauptmann, CMU
Milind R. Naphade, IBM
Columbia University
Research Alan Smeaton, Dublin
Ramesh Jain, Georgia Edward Chang,
City University
Institute of Technology UCSB
HongJiang Zhang,
Thomas Huang, UIUC
Nevenka Dimitrova,
Microsoft Research
Edward Delp, Purdue
Phillips Research
University Ajay Divakaran, MERL
Rainer Lienhart, Intel
Wessel Kraaij,
Apostol Natsev, IBM
Information Systems
Research
Division, TNO TPD
Tat-Seng Chua, NUS
Ching-Yung Lin, IBM
Ram Nevatia, USC
Research
John Kender,
Mubarak Shah,
Columbia University
University of Central
Florida
Page
PNN MITRE
97. Impact and Outcome
• First of a Kind Ontology of 1000 or more semantic concepts that have been
evaluated for their usability and feasibility by different communities including UC,
OC, MC.
• Annotated corpus (200 hours) and ontology can be further exploited for future
TRECVID, VACE, MPEG-7 activities. Core semantic primitives, that can be included
in various video description standards/languages such as MPEG-7.
• Empirical and theoretical study of automatic concept detection performance for
elements of this large ontology. Use of current state of the art detection wherever
possible. Use of simulation where the detection is not available.
• Use cases (queries) testing and expansion into ontology
• Reports documenting use cases, existing practices, research challenges and
recommendations
• Prototype systems and tools for annotation, query formulation and evaluation
• Guidelines on manual and automatic multimedia query formulation techniques going
from use-cases to concepts.
• Categorization of classes of concepts based on feasibility, detection performance
and difficulty in automation
BOTTOMLINE: All this is driven by the user
Page
PNN MITRE
98. Summary of Key Questions
• How easy was it to create annotations
- (man-hours/hr of video?)
• How well does the lexicon 'partition' the collection
• Given perfect annotations/classification:
- How well does the lexicon aid with queries/tasks
• How good is automatic annotation of the sample collection
- What fraction of perfect annotations accuracy is obtained for the
queries/tasks
• How much is automatic classification performance of a given lexical item
a function of training data
- Estimate how much training data would get this lexical item to 60%,
80%, 90%, 95%?
• What lexicon changes are necessary or desirable?
Page
PNN MITRE
99. Video Event Ontology (VEO) & VEML
• A Video Event Ontology was developed in the ARDA workshop on
video event ontologies for surveillance and meetings allows natural,
hierarchical representation of complex spatio-temporal events
common in the physical world by a composition of simpler (primitive)
events
• VEML – XML-derived Video Event Markup Language used to annotate
data by instantiating a class defined in that ontology. Example: We
will attempt to use or adapt their notation to the extent possible
• (http://www.veml.org:8668//space/2003-10-08/StealingByBlocking.veml)
• Broadcast video news ontology is likely to have little overlap with the
complex surveillance events described in the VEO, except for some
basic concepts. We expect our ontology to be broader, but much
shallower
• Our broadcast news ontology is largely applicable to any edited
broadcast video (e.g. documentaries, talk shows, movies) and
somewhat applicable to video in general (including surveillance, UAV
and home videos).
Page
PNN MITRE