The slides from our presentation at the Symposium on Software Performance in Munich (November 2015) presenting the diagnoseIT project on diagnosing software performance problems. It gives also a brief insight into the Common Trace API to access monitoring data captured by tools of different vendors.

1. Fighting Groundhog Days:
Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
Symposium on Software Performance (SSP)
November 05, 2015 @ Munich
Christoph Heger, Dušan Okanović, Stefan Siegl, André van Hoorn and Alexander Wert
@diagnoseIT_apm https://diagnoseIT.github.io
(http://goo.gl/Q3OZpA)

2. 2/ 2015/11/05
Performance Problems are Omnipresent
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
An unexpected
error occured
Temporarily
not available
… more capacity
is on the way
Try again later
Please visit us
again later
We are
experiencing
heavy demand

3. 3/ 2015/11/05
Manual Diagnosis
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
Detect problem

4. 4/ 2015/11/05
Manual Configuration
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
Adapt
instrumentation

5. 5/ 2015/11/05
Manual Reasoning
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
Component
Deep-Dive
Interpretation of
measurements

6. 6/ 2015/11/05
diagnoseIT Goals
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
Reduce
maintainance
effort by 50%
Detect 100% of
all problems
automatically
Identify 80% of
the root causes
automatically
No APM
vendor lock-in

7. 7/ 2015/11/05
Overview
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
Node 1
JVM
App 1
App 2
Node n
SUT
Traces
CTA
Instrumentation
Refinement
Request
Instrumentation
Language
Result
Result Description
Language
Monitoring Tool
Dynamic
Instrumentation
API Labelling
Location
Identification
Instrumentation
Quality Manager
Result Query

8. 8/ 2015/11/05
Diagnosis
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
Trace
Traces
CTA
Rule
Retrieve
information from
Notify on tagging
Problem
Instance
Repository
Manage
Tag
Instrumentation Refinement Request
Instrumentation
Language
Insight
Enrichment
Information
Request
Maintain
Problem
Instance
Problem
Instance
Create/Update
Result Description
Language
MonitoringTool

9. 9/ 2015/11/05
diagnoseIT Characteristics
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
Adaptive
Configuration
Continuous
Learning
APM Process
Automation
Formalized
Expert
Knowledge

10. 10/ 2015/11/05
Rules
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
Root
Cause
Problem
Context
Global
Context
…
Root Cause
Tag
Problem
Context Tag
Global
Context Tag
…
…
…
Isolate
Root Cause
Semantify
Root Cause
Generic Analysis
Domain- and Technology-
specific Analysis

11. 11/ 2015/11/05
DVDStore Example
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
doFilter(…) 47.67 186,259.50
searchTitleAndDescription(…) 22,961.16 184,897.25
searchTitleAndDescriptionWithOneWord(…) 0.14 158,054.08
list(…) 152,620,13 158,036.29
executeQuery() 1,793.75 1,793.75
executeQuery() 0.36 0.36
…
executeQuery() 0.13 0.13
searchTitleAndDescriptionWithOneWord(…) 0.05 3,882.01
list(…) 2,625.22 3,881.70
executeQuery() 1,257.47 1,257.47{SQL: Select PROD_ID …}
{SQL: Select PROD_ID …}
Real Trace
(Simplified)
{SQL: Select INV_ID …}
{SQL: Select INV_ID …}
Response Time
[ms]
Exclusive Execution
Time [ms]

12. 12/ 2015/11/05
Root Cause Isolation
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
list(…) 152,620,13 158,036.29
searchTitleAndDescription(…) 22,961.16 184,897.25
list(…) 2,625.22 3,881.70
executeQuery() 1,793.75 1,793.75
executeQuery() 1,257.47 1,257.47
doFilter(…) 47.67 186,259.50
executeQuery() 0.36 0.36
searchTitleAndDescriptionWithOneWord(…) 0.14 158,054.08
executeQuery() 0.13 0.13
searchTitleAndDescriptionWithOneWord(…) 0.05 3,882.01
Real Trace
(Simplified)
Response Time
[ms]
Exclusive Execution
Time [ms]
{SQL: Select PROD_ID …}
{SQL: Select PROD_ID …}
{SQL: Select INV_ID …}
{SQL: Select INV_ID …}
Descending Order
Cause
Cause
Cause
Cause
Cause

13. 13/ 2015/11/05
Problem Instance 1
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
CP
doFilter(…) 47.67 186,259.50
searchTitleAndDescription(…) 22,961.16 184,897.25
searchTitleAndDescriptionWithOneWord(…) 0.14 158,054.08
list(…) 152,620,13 158,036.29
executeQuery() 1,793.75 1,793.75
executeQuery() 0.36 0.36
…
executeQuery() 0.13 0.13
searchTitleAndDescriptionWithOneWord(…) 0.05 3,882.01
list(…) 2,625.22 3,881.70
executeQuery() 1,257.47 1,257.47{SQL: Select PROD_ID …}
Cause
{SQL: Select PROD_ID …}
Real Trace
(Simplified)
Problem Context
{SQL: Select INV_ID …}
{SQL: Select INV_ID …}
Response Time
[ms]
Exclusive Execution
Time [ms]

14. 14/ 2015/11/05
Problem Instance 2
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
CP
doFilter(…) 47.67 186,259.50
searchTitleAndDescription(…) 22,961.16 184,897.25
searchTitleAndDescriptionWithOneWord(…) 0.14 158,054.08
list(…) 152,620,13 158,036.29
executeQuery() 1,793.75 1,793.75
executeQuery() 0.36 0.36
…
executeQuery() 0.13 0.13
searchTitleAndDescriptionWithOneWord(…) 0.05 3,882.01
list(…) 2,625.22 3,881.70
executeQuery() 1,257.47 1,257.47
Cause
{SQL: Select PROD_ID …}
Cause
{SQL: Select PROD_ID …}
Real Trace
(Simplified)
Problem Context
{SQL: Select INV_ID …}
{SQL: Select INV_ID …}
Response Time
[ms]
Exclusive Execution
Time [ms]

15. 15/ 2015/11/05
Problem Instance 3
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
CP
Response Time
[ms]
doFilter(…) 47.67 186,259.50
searchTitleAndDescription(…) 22,961.16 184,897.25
searchTitleAndDescriptionWithOneWord(…) 0.14 158,054.08
list(…) 152,620,13 158,036.29
executeQuery() 1,793.75 1,793.75
executeQuery() 0.36 0.36
…
executeQuery() 0.13 0.13
searchTitleAndDescriptionWithOneWord(…) 0.05 3,882.01
list(…) 2,625.22 3,881.70
executeQuery() 1,257.47 1,257.47Cause{SQL: Select PROD_ID …}
Cause{SQL: Select PROD_ID …}
Real Trace
(Simplified)
Problem Context
{SQL: Select INV_ID …}
{SQL: Select INV_ID …}
Exclusive Execution
Time [ms]

16. 16/ 2015/11/05
Global Context
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
Response Time
[ms]
doFilter(…) 47.67 186,259.50
searchTitleAndDescription(…) 22,961.16 184,897.25
searchTitleAndDescriptionWithOneWord(…) 0.14 158,054.08
list(…) 152,620,13 158,036.29
executeQuery() 1,793.75 1,793.75
executeQuery() 0.36 0.36
…
executeQuery() 0.13 0.13
searchTitleAndDescriptionWithOneWord(…) 0.05 3,882.01
list(…) 2,625.22 3,881.70
executeQuery() 1,257.47 1,257.47{SQL: Select PROD_ID …}
{SQL: Select PROD_ID …}
Real Trace
(Simplified)
{SQL: Select INV_ID …}
{SQL: Select INV_ID …}
Exclusive Execution
Time [ms]
3x Problem Context Global Context

17. 17/ 2015/11/05
diagnoseIT Prototype
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
Problems
Overview
Natural language
description
Problem instance
details

18. 18/ 2015/11/05
diagnoseIT Prototype
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
Problem
Highlighting

19. 19/ 2015/11/05
Common Trace API
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
https://goo.gl/UuoZkN
Join us
APM vendor independent data access
Goal
Monitoring data format is tool-specific
Motivation
APM vendor lock-in limits
interoperability and data exchange
Drawback
Importers
Planned
(http://goo.gl/5S337v)
(http://goo.gl/AAwdge)
(https://goo.gl/eDuA9R)
(http://goo.gl/3xM2ur) (http://goo.gl/KCWimU)
(http://goo.gl/z18Tuj) (http://goo.gl/B3rJt4)

20. 20/ 2015/11/05
diagnoseIT-as-a-Service
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
Evaluation of diagnosis rules
Motivation
Diagnosis service for traces
Idea
Collection of real traces and problems
Benefit
Support us
https://ditaas.inspectit.rocks

21. 21/ 2015/11/05
Future Steps
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
Anti-Pattern
Detection
Overhead-aware
Instrumentation
Case Studies
System
measures

22. 22/ 2015/11/05
Case Studies Outlook
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
Level of Distribution
LevelofComplexity
DVDStore
(http://goo.gl/dPc0YQ)
(http://goo.gl/yDHkW6)
(http://goo.gl/8PnaqE)
(https://goo.gl/oqvy4K)

23. 23/ 2015/11/05
Conclusion
Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of
Performance Problems in Enterprise Applications
Demo Bundle
https://goo.gl/M1VQA9@diagnoseIT_apm https://diagnoseIT.github.io
Problem Detection
Automatic Diagnosis
APM Process
Automation
No APM
vendor lock-in