1. Introduction Dimensions Results
Comparison of Reasoners for large Ontologies in
the OWL 2 EL Profile
WAI
Kathrin Dentler
November 8th 2010
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 1/26

2. Introduction Dimensions Results
Outline
1 Introduction
2 Dimensions
3 Results
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 2/26

3. Introduction Dimensions Results
Outline
1 Introduction
2 Dimensions
3 Results
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 3/26

4. Introduction Dimensions Results
PhD Project
Supervisors
Ronald Cornet Annette ten Teije
Nicolette de Keizer Frank van Harmelen
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 4/26

5. Introduction Dimensions Results
PhD Project
SEMI4
SEMantic Interoperability for Integration of clinical
Information and Indicators
Quality Indicators
General Problem: growing number, labour intensive
Approach:
formalize indicators;
integrate data sources;
investigate which adjustments are needed to facilitate
data-capture at the required quality levels
→ More in next WAI
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 5/26

6. Introduction Dimensions Results
SNOMED CT
Systematized Nomenclature of Medicine, Clinical Terms
SNOMED CT
used to represent information in electronic health records
around 300,000 active concepts
available as DB tables and as OWL ontology
in tractable OWL 2 profile EL
Goal: Classify SNOMED CT with a reasoner to obtain the
closure, i.e. all possible inferences.
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 6/26

7. Introduction Dimensions Results
An example OWL 2 EL ontology
Fracture Traumatic abnormality by morphology
Traumatic abnormality by morphology Traumatic abnormality
Traumatic abnormality Damage
Fracture of foot ≡ Fracture of lower limb ∃.rolegroup
(∃associated morphology.Fracture
∃finding site.Bone structure of foot)
Fracture of lower limb ≡ Fracture of bone ∃rolegroup.
(∃associated morphology.Fracture
∃finding site.Bone structure of lower limb)
Inferred:
Fracture Damage
Fracture of foot Fracture of lower limb
Pellet’s explanation:
Fracture of foot ≡ Fracture of bone ∃.rolegroup
(∃associated morphology.Fracture
∃finding site.Bone structure of foot)
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 7/26

8. Introduction Dimensions Results
Included Reasoners
CB
CEL
FaCT++
HermiT
Pellet
Racer
Snorocket
TrOWL
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 8/26

9. Introduction Dimensions Results
Related Work
Earlier Comparisons of Classification Performance for SNOMED CT
10000 CB
CEL
FaCT
Racer
Snorocket
Classification Time in Seconds (log-scale)
1000
100
10
2005 2006 2008 2009 2010
Timeline
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 9/26

10. Introduction Dimensions Results
Outline
1 Introduction
2 Dimensions
3 Results
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 10/26

11. Introduction Dimensions Results
Dimensions
1 Reasoning Characteristics
2 Practical Usability
3 Performance Indicators
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 11/26

12. Introduction Dimensions Results
Dimension Reasoning Characteristics
Independent of Ontology
Methodology
Soundness and Completeness in Theory
Expressivity and Computational Complexity
Incremental Classification
Rule Support
Justifications
Support of ABox Reasoning Tasks
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 12/26

13. Introduction Dimensions Results
Dimension Practical Usability
Independent of Ontology
OWL API
OWLlink
Availability as Prot´g´ Plugin
e e
License
Open or Closed Source
Programming Language
Supported Platforms
Jena
Kind of Institution
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 13/26

14. Introduction Dimensions Results
Dimension Performance Indicators
Related to Ontology
Classification Performance
TBox Consistency Checking Performance
Concept Satisfiability Checking Performance
Subsumption Query Performance
Required Heap Space
Soundness and Completeness in Practice
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 14/26

15. Introduction Dimensions Results
Outline
1 Introduction
2 Dimensions
3 Results
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 15/26

16. Introduction Dimensions Results
Results: Categorization of Reasoners
Dimension Reasoning Characteristics
CB CEL FaCT++ HermiT Pellet RP SR TrOWL
(REL)
Methodology consequ.- compl. tableau- hyper tableau tableau compl. approx.
based rules based tableau based based rules (compl.
rules)
Sound
( )
Complete
x( )
IC
(addition/ x/x /x x/x x/x / x/x /x x/x
removal)
Rule x x x x x
Support
(SWRL) (SWRL) (SWRL,
nRQL)
Justifications x x x x x
ABox x x
Reasoning
(SPARQL) (SPARQL, (SPARQL)
nRQL)
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 16/26

17. Introduction Dimensions Results
Results: Categorization of Reasoners
Dimension Practical Usability
CB CEL FaCT++ HermiT Pellet RP SR TrOWL
OWL API x
OWLlink API x x x
Prot´g´ Plugin
e e x x
License DuLi: GLGPL AP 2.0 GLGPL GLGPL DuLi: AGPL own own DuLi: AGPL
Open Source x x x
Language OCaml Common Lisp C++ Java Java Lisp Java Java
Platforms all Linux all all all all all all
Jena x x x x x x x
Institution a a a a c c g a
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 17/26

18. Introduction Dimensions Results
Results: Categorization of Reasoners
Dimension Performance Indicators
Experiments on Galjas
With: (GO, NCI and) SNOMED CT
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 18/26

19. Introduction Dimensions Results
Classification Performance vs. Supported Expressivity
Dimension Performance Indicators
Supported Expressivity
EL EL EL EL Horn SHIQ SHIQ(D-) SROIQ(D) SROIQ(D)
G0
NCI
SNOMED CT
1000
Classification Time in Seconds (log-scale)
100
10
1
0.1
CEL Pellet Snorocket TR CB RacerPro HermiT FaCT++
Reasoners ordered by increasing Supported Expressivity
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 19/26

20. Introduction Dimensions Results
Consistency Checking Performance
Dimension Performance Indicators
CB CEL FaCT++ HermiT Pellet RP SR TR
OSNOMED CT x 0.88 15.3 0.00 16.78 x 0.00 0.00
CEL and TrOWL REL need to classify first, but return true
anyway, Snorocket outputs a warning.
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 20/26

21. Introduction Dimensions Results
Concept Satisfiability Checking Performance
Dimension Performance Indicators
CB CEL FaCT++ HermiT Pellet RP SR TR
OS CT BC x 38.42 22.37 5,276.85 56.91 273.45 0.07 5.17
OS CT AC x 34.59 1.76 1.36 6.07 0.00 0.06 0.46
E.g TrOWL returns reliable results only after the ontology is
classified.
→ benchmarks with inconsistent ontologies / unsatisfiable
concepts needed!
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 21/26

22. Introduction Dimensions Results
Subsumption Query Performance
Dimension Performance Indicators
CB CEL FaCT++ HermiT Pellet RP SR TR
NC BC
# results x 1 20 20 20 20 x 0
seconds x 0.96 701.79 6,649.85 2,793.31 3,380.67 NPE 0.17
AC BC
# results x 1 1 20 20 20 x 0
seconds x 0.00 0.06 16.94 0.49 0.74 NPE 0.00
NC AC
# results x 20 20 20 20 20 20 20
seconds x 0.00 0.00 0.00 0.00 0.70 0.00 0.28
AC AC
# results x 1 1 20 20 20 20 0
seconds x 0.00 0.06 17.12 0.00 0.92 6.97 0.00
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 22/26

23. Introduction Dimensions Results
Minimum Heap Space for Java Reasoners
Dimension Performance Indicators
CB CEL FaCT++ HermiT Pellet RP SR TR
n/a n/a n/a 4.5 10 n/a 2.5 4
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 23/26

24. Introduction Dimensions Results
Soundness and Completeness in Practice
Dimension Performance Indicators
CB: missing SubClassOf owl:Thing for top-level classes
CEL: additional EquivalentClasses and EquivalentProperties
for each class and property
RacerPro: additional sub class owl:Nothing for all leaf concepts
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 24/26

25. Introduction Dimensions Results
Soundness and Completeness in Practice
Dimension Performance Indicators
Table: SNOMED CT: Missing / Additional inferred SubClassOf
statements in regard to the Relationships Table
CB CEL FaCT++ HermiT Pellet RP SR TR
Missing 0 0 0 0 386 0 86 0
Additional 0 0 0 0 546 0 34 0
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 25/26

26. Introduction Dimensions Results
Questions ?
Kathrin Dentler
k.dentler@amc.uva.nl
k.dentler@few.vu.nl
http://reasonerbench.net/
Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 26/26