Sascha Seifert, Siemens Healthcare, pour la journée e-health 2013

Answers for life.Unrestricted © Siemens AG 2013 All rights reserved.
Theseus Medico and
imaging in the digital diagnosis
Dr. Sascha Seifert
eHealth Day
Sierre
June 2013

Unrestricted © Siemens AG 2013 All rights reserved.
Page 2 Dr. Sascha Seifert / Healthcare, Imaging & Therapy, SYNGO
THESEUS-MEDICO consortium
2
Fraunhofer
IGD
Siemens CT
Erlangen &
Munich
Averbis
Ludwig-Maximillian
University Munich
University
Hospital
Erlangen
Transinsight
DFKI
• 11/2007 – 05/2012
• 67.5 man-years,
• grant 50% government, 50% industry
• 3 research centers, 1 hospital
• 3 companies (Siemens & 2 SME)
• Lead with Siemens AG

Codes
Control
1980 - 2000
physics
Robust Learning Methods
Semantic Web Standards
2000-2010
data
Open Internet databases
Business intelligence
Big Data analytics
Semantic interoperability
2010 - future
content
Quelle: Gartner. Hype Cycle for Healthcare Provider Technologies and
Standards, July 2010
Theseus-Medico
(Semantic Web for Medicine)
Google „understands“ now context; knowledge graph with 570 million elements, 18 billion
facts, launched in 2012
RSNA.org drives semantic web
standard for radiology
Evolution of medical data processing
Comprehending Software

The THESEUS-MEDICO approach
Semantic Web1Radiologist / Clinician
• Content Understanding
• Content Linking
• Content Search
• Knowledge explosion
• Internet, books, articles
• Data overload
• Unstructured data: Images, Texts
• Structured data: Lab values, Medication 1by Tim Berners-Lee

Application Prototype (National IT-Summit 2011)

Text Mining FormalizationTextMiningSegmentation Formalization
SemanticProcessingofMedicalData
Medical Images Radiology Reports Treatment Plans Online Knowledge Expert Knowledge
Medical Image
Annotation Representational Ontology: OWL
Upper Ontology:
time, space, organization, person, event
moredomainspecific
morelikelytobechanged
Information Element
Ontology
images, texts,
volumes, …
AnnotationOntology
Clinical Ontology
-doctor, nurse, patient
-medical case
-DICOM Ontology
mid-level
ontology
low-level
ontology
Medical Ontologies
FMA
ICD-10
Website
RadLex
FMA
Exten.
Mappings
toexternal
sources
ICD-10
mapping&
merging
Visual
Charac.
Mappings
toexternal
sources
annotation
Thesauri & Taxonomies
extraction
Disease-Symptom
Navigation
Multi-Modal
Interaction
Quality Control Intelligent Diagnose
Ontology EngineeringInformation Extraction
from Medical Texts
Semantic Search Semantic Reporting Image & Text Linkage
Intelligent Healthcare Applications
Intelligent applications using knowledge services
Knowledge Services Infrastructure
Knowledge Extraction and Formalization
Knowledge Management Overview
Clinical Workflow Components
2,5 TB of 750 patients
(≙ ~ 7000 series) 6000 rad reports

IMAGE AND TEXT UNDERSTANDING

Support systems for disease recognition
Anatomical Body Regions and Organs
Lungs, heart, liver, spleen, kidneys, prostate, urinary bladder,
esophagus, pancreas and several anatomical landmarks
Valve function, coroanry stenosis, osteolytic tumors, liver
tumors, lymph node cancer
ImageParsing

Increase of knowledge about the image content
Image
Content
Detection Segmentation
Prior knowledge
# Images 3-fold C.V. [mm] Runtime [s/vol]
Heart* 457 1,30 3,55
Liver 346 1,07 6,00
Spleen 203 2,14 9,90
Right kidney 199 1,03 0,40
Left kidney 197 1,15 0,40
Left lung 166 2,64 1,70
Richt lung 163 2,35 1,80
Urinary bladder 141 1,35 1,00
8 organs, 19 landmarks,
3 body regions <1 min (09/2010)
Generic Image Parsing

Knowledge based image processing for anatomy understanding
Mean error 1,80±1,17 mm
Mean error 1,70±0,71 mm
Panceas segmentation considering splenic vein
i
Thoracic lymph nodes
Esophagus

Osteoblastic Osteolytic
Total Volumes 30 20
Total Annotations 172 42
False Positives per Patient 3.5 3.7
Overall Sensitivity 83% 88%
Mean Sensitivity 80% 93%
Overall Positive Predictive Value 58% 35%
Mean Positive Predictive Value 65% 49%
+9 monthsBaseline
Th8
osteolytic
volume 10%
lower end-
plate
Th8
mixed
volume 95%
whole vertebra
Fracture
progression
Th11
osteolytic
volume 28%
vertebra back
new
Prior knowledge: vertebrae / discs
Constrains lesions search and automatically find
corresponding lesions in prior exams
Semantic detection and follow-up of spine lesions for bone lesions
Results may vary. Data on file

Semantic detection of coronary findings
• Automatic vessel tracing and labeling of coronary
vessels (Random forest)[Gülsün2008]
• Automatic estimation of lumina by Random Forest
Regressor (10x faster than segmentation)
• Characteristics curves along vessel (e.g., degree of
calcification)
• Identification of potential stenosis
• Additional information such as FFR
(measured/simulated) is semantically linked
calcified
non-
calcified mixed overall
by
lesion
sensitivity 96.55% 89.23% 91.78% 94.75%
FPR 1.50 2.30 0.87 4.67
by
vessel
sensitivity 98.67% 94.44% 92.16% 96.47%
specificity 79.12% 54.35% 81.39% 71.27%
NPV 99.58% 99.21% 99.30% 99.37%
10-fold cross validation with 256 CCTA volumes runtime <2 min / volume
Knowledge pipeline:
centerlinelumenstenosisclassification
Accelerate and quantify readingResults may vary. Data on file

Vessel
MEDICO 14
… use knowledge about the anatomy (shape) of
the liver to register accurately.
Registration of multi modal and multi phase examinations
Instead of just comparing grey-values…

Page 15 Dr. Sascha Seifert / Healthcare, Imaging & Therapy, SYNGOMEDICO 15
Registration of multi modal and multi phase examinations

Page 16 Dr. Sascha Seifert / Healthcare, Imaging & Therapy, SYNGOMEDICO 16
Fusion view of head-neck tomographies
Segmenting bones Rigid bone registration “Adherence” of soft tissue
Using knowledge about anatomy of bones and soft tissue

Semantic Computer Aided Detection
• Horizontal and vertical integration of computer aided detection
• Disadvantage of current solutions:
• Specific sub systems
w/o information exchange /
consolidation
• „Syntactic“ interoperability
Reporting
Semantic
Search
Bone lesion CAD
Breast CAD
Liver CAD
Patient context
Bone (e.g. Spine) segmentation
THESEUS-MEDICO:
Standardized, semantic information
using common or (machine-) convertible
vocabulary

Three Information Dimensions
of Radiology Reports - Challenges
1 Anatomical / Spatial information
• Location of a finding, e.g. affected organ, lymph nodes
• Spatial modifier, e.g. left, right, axilliary
2 Pathological Information
• Pathological interpretation of the highlighted finding,
e.g. size (enlarged lymph nodes), density (lung nodule),
number of occurrences, etc.
3 Temporal Information
• Provides information about the difference/changes
of the current findings in relation to past findings, e.g.
In comparison to prior examination…
Compound words (German): Stemming
• Split into sub words before semantic mapping to reduce solution space
• Data reduction of 90% in German language
Example:
Unchanged, not pathologically enlarged axilliary,
mediastinal and hilar lymph nodes.
FINDINGS:CHEST (EXAMPLE)
The lungs demonstrate bilateral areas of pulmonary consolidation, involving
predominantly the right upper lobe and to a lesser extent the left upper lobe posterior
lingula and superior segment of the lower lobes with additional patchy opacities in the
right lung base and right middle lobe. Findings are compatible with multi lobar
pneumonia. There are bilateral right greater than left pleural effusions, small in size.
There is diffuse anasarca present with 3rd spacing of fluid. There is an old healed
displaced right clavicle fracture noted. There are sub centimeter hypodense nodules in
both thyroid lobes. There are sub centimeter lymph nodes in the mediastinum measuring
up to 9mmin size in the precarinal space, probably reactive. There are calcified lymph
nodes in the hila bilaterally, from prior granulomatous disease. There is bibasilar passive
atelectasis adjacent to the effusions, with calcified granulomata in both atelectatic lower
lobes. The heart is globally enlarged, with coronary artery, aortic valve calcifications
present. Additional calcified granulomata are shown in the anterior segment of the right
upper lobe. The main pulmonary artery segment is dilated up to 3.3cm in size,
suggesting pulmonary artery hypertension.
Myo|kard|itis
Herz|muskel|entzünd|ung
Inflamm|ation of the heart muscle
muscle
myo
muskel
muscul
inflamm
-itis
inflam
entzünd
KONZEPT
subwort herz
heart
card
corazon
card
INFLAMMATION
MUSCLE
HEART

100 report texts from lymphoma
cases were manually annotated
RadMiner™ Report Search, Averbis GmbH
Averbis Annotator
(internal)
Validation
Precision=0.921
Recall=0.935
F1=0.928
Orig: Radlex 2.0
Stem: with stemming
Sem: Radlex extended
Semantic Report Search
Natural Language Processing + Semantic Mapping
Results may vary. Data on file

Knowledge Representation
Represent segmentations with semantics
Manuel Möller, DFKI Kaiserslautern, manuel.moeller@dfki.de
Semantic
Annotation
Medical
Ontology
Link by
Semantic
Concepts
from
Ontology

Annotation
Ontology
Consolidated data model: knowledge exchange instead of data exchange using
semantic web standards (RDF and OWL)
new* SPARQL 1.1 (recursive queries)
*W3C Working Draft 05 January 2012
Ontology Terms
Snomed CT 395036
FMA 83281
Radlex 34895
Reference Ontologies
Content
Physical
Ref.
Physical
Ref.
Reference
Ontologies
Representation Language
The whole picture

SEMANTIC READING

Semantic Reading – Compare Studies based on Anatomy instead of
Frame of Reference
Semantics enables registration as expected by radiologists.
Locating corresponding
anatomical structures
by concepts
Anatomy is compared with
the same anatomy
Benefits
•synchronized scrolling
•compare findings over
time
•compare similar patients
bronchial
bifurcation

Text-to-Image Linking
(Navigation support)
• Hyperlinks encode semantic information and enable to
bidirectional jumps
• Close semantic gap between image and text based
systems.
• Improve dialogue of radiologist and clinicians (ideal for
radiol. demonstrations, easy access of priors)
Anatomy aware findings labeling
• Benefit from image understanding.
• More meaningful findings names,
understandable by computers
• Enables the system to infer knowledge
Semantic Reading – Intelligent applications

Anatomy aware literature references
Image understanding enables to display literature
references matching the underlying anatomy, e.g. the
Bosnial classification for renal cysts.
Example subjects:
• Bosniak renal cyst classification
• Fleischner pulmonary nodule management
• Couinaud liver segments
• Lung segments
• Hydronephrosis classification
• TNM
Semantic lesion progression
Related findings retrieved with
semantic reasoning
Semantic Reading – Intelligent applications

DISEASE MODELING

clinical data
Medical Images
Reports
lymph node of head
Lingual lymph node
mandibular lymph node
lymph node
lymph node of trunk
malar lymph node
size modifier
shrunken Annotations
facial lymph node
enlarged
MEDICO-Annotation-Ontology
WhatweareusingOurextension
Use of external knowledge
lymphoma
hasLeadingSymptom
symptom
disease
is-a
is-a
enlarged lymph node
Disease-Symptom-Ontology
DiSy:hasModifier
DiSy:locatedIn
Inferlikelydiseases
Clinical Recommendations
Next Examinations
From Annotations to Diagnosis

Differential
Diagnosis
Definitions
Probabilities
Correlation
Source: Herold, Innere Medizin,
2011
Analysis of available Clinical Knowledge

5 Diseases
Hodgkin-Lymphoma
Non-Hodgkin-Lymphoma
Correctal Carcinoma
Reactive Lymphadenitis
40 Symptoms
If possible with definition and
probabilities
Encoding with
RadLex or SNOMED CT
10 Dummy Patients
Information about Leading
Symptoms
Test Data Set provided by experts

Ranking factors:
 Age, gender, specific incidence
 Leading symptoms
 Symptom intensity
 Reappearing symptoms
 Relative importance of symptoms
 Ratio of present and absent symptoms
of a disease
Towards a Ranking of Likely Diseases in Terms of Precision and Recall Heiner Oberkampf, Sonja Zillner, Bernhard Bauer, Matthias Hammon, Netmed2012.
Ranking of Likely Diseases

Ranked symptoms list is provided
Recommendations of next examinations
Decision Support

SEMANTIC SEARCH

Medical images can only be searched using:
• meta data in so-called DICOM-headers
(patient name, acquisition date, imaging modality etc.)
• indirectly by searching corresponding radiology reports
‘Content’ of the images can not be
used for
• quality control
• data mining for clinical /
epidemiological studies
• decision support
• workflow improvements
• reporting support
Semantic Limitations of Today's Hospital IT
wrt Medical Images

„Find patients with similar liver lesion, enlarged (pathological) lymph nodes in the thorax, hemoglobin value low and
patient age greater than 65“
Resulting patients
with
similar findings
Reference
lesion
Findings histogram
for query
refinement
Accessing
reports and lab
values
Query
terms
Integrated Semantic Image Search
PET-CT 32%
MRI 12%
Acquisition methods
CHOP14
DXBEAM_C
Treatments
IMVP16
Germany
• DSHNHL2004-2 (FLYER) Phase 3
• DSHNHL2006-1B (ACT-2) Phase 3
• DSHNHL2002-1 (Mega-CHOEP) Phase 3
USA
• UCLA-0406049-01 Phase 3
Clinical trials

„Search for images and texts
of patients with thickened wall
of intestine and
hemoglobin value low“
(cancer?)
Including lab values
Understanding the anatomy:
intestine expands to rectum, colon,
sigmoid, cecum, …
Full text search is not enough!
Integrated Semantic Image Search

Image Retrieval - Results
• 111 liver lesion with 6105 pairs annotated according to similarity
• Annotated with 5 similarity levels; Leave-One-Out validation
MAP: 0.78
nDCG(10): 0.85
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0.000.250.500.751.00
Recall
Precision
search
input
output
Hammon, M.; Dankerl, P.; Costa, M.; Tsymbal, A.; Seifert, S.; Sühling, M.;
Uder, M. & Cavallaro, A. (2012), Computer-aided decision support for the
characterization of liver lesions in CT scans, in 'Proceedings of the
European Congress of Radiology (ECR)‘.

• transform data into actionable knowledge by image, text and speech understanding
• harmonize CAD by using standardized vocabulary (CAD group, SCR, and 3rd party)
• improve interoperability by describing content in a standardized (semantic web) way
• enable inter-modal semantic navigation, i.e. between image, text and other clinical data
• Understanding content is the key for proactive context-sensitive workflow support
• organize information semantically and prepare for data analytics (understanding first)
• formalize medical knowledge instead of programming
• enable reuse of third party knowledge networks / databases (publishers, clinical trials, ICD10, ….)
Summary

Dr. Sascha Seifert
H SY TI / Germany
Hartmannstr. 16
91052 Erlangen
E-mail:
saschaseifert@siemens.com
Contact
Disclaimer:
The MEDICO prototypes are under development and not commercially available, and their
future availability cannot be ensured. The prototypes should not be used for any patient
diagnosis or therapy. MEDICO is not related to the commercial hospital information system
Medico.
Acknowledgements:
The MEDICO project is supported in part by the THESEUS program, which is funded by the
German Federal Ministry of Economics and Technology under the grant number
01MQ07016. The responsibility for this demonstration lies with the authors.

syngo. It’s all about you.

Sascha Seifert, Siemens Healthcare, pour la journée e-health 2013

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Andere mochten auch

Andere mochten auch (14)

Ähnlich wie Sascha Seifert, Siemens Healthcare, pour la journée e-health 2013

Ähnlich wie Sascha Seifert, Siemens Healthcare, pour la journée e-health 2013 (20)

Mehr von Thearkvalais

Mehr von Thearkvalais (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Sascha Seifert, Siemens Healthcare, pour la journée e-health 2013