jndfjnj

1. Anomaly Detection
Softnix Technology
Chakrit Phain

2. Agenda
2
• Part 1
• Introduction
• Application for Anomaly Detection
• AIOps
• GraphDB
• Part 2
• Type Of Anomaly Detection
• How to Identify Outliers in your Data
• Part 3
• Anomaly Detection for Timeseries Technique

3. Introduction
3
https://en.wikipedia.org/wiki/Anomaly_detection
In data mining, anomaly detection (also outlier detection[1]) is the
identification of rare items, events or observations which raise suspicions by
differing significantly from the majority of the data.

4. Introduction
4
https://www.anodot.com/blog/what-is-anomaly-detection/
https://developer.mindsphere.io/apis/analytics-anomalydetection/api-anomalydetection-overview.html

5. Introduction
5

6. Introduction
6

7. Applications of Anomaly Detection in the Business
world
7
• Intrusion detection
• Attacks on computer systems and computer networks.
• Fraud detection
• The purchasing behavior of some one who steals a credit card is probably
different from that of the original owner.
• Ecosystem Disturbance.
• Hurricanes , floods , heat waves … etc
• Medicine.
• Unusual symptoms or test result may indicate potential health problem.
https://zindi.africa/blog/introduction-to-anomaly-detection-using-machine-learning-with-a-case-study

8. Introduction
8
https://www.datasciencecentral.com/profiles/blogs/driving-ai-revolution-with-pre-built-analytic-modules

9. AIOps

10. AIOps
10
https://aiops.fatpipes.biz/2019/09/aiops-best-for-anomalies-noise-alert.html?utm_source=dlvr.it&utm_medium=twitter

11. AIOps
11

12. AIOps
12

13. AIOps
13
https://infrastructureedge.blogspot.com/2019/09/enterprise-it-managers-say-aiops-works.html?utm_source=dlvr.it&utm_medium

14. AIOps
14

15. GRAPH DATABASE

16. Fraud Detection for Graph analytic
16
https://neo4j.com/whitepapers/fraud-detection-graph-databases/?ref=blog

17. Insurance Fraud
17
https://neo4j.com/whitepapers/fraud-detection-graph-databases/?ref=blog

18. Fraud Prevention Approach
18

19. Type Of Anomaly

20. Anomalies can be broadly categorized as
20
1. Point Anomalies
2. Contextual Anomalies
3. Collective Anomalies
If an individual data instance can be considered as
anomalous with respect to the rest of data, then the instance is
termed as a point anomaly.
Chandola, V., Banerjee, A. & Kumar, V., 2009. Anomaly Detection: A Survey. ACM Computing Surveys, July. 41(3).

21. Anomalies can be broadly categorized as
21
1. Point Anomalies
2. Contextual Anomalies
3. Collective Anomalies
If a data instance is anomalous in a specific context
(but not otherwise), then it is termed as a
contextual anomaly
Chandola, V., Banerjee, A. & Kumar, V., 2009. Anomaly Detection: A Survey. ACM Computing Surveys, July. 41(3).

22. Anomalies can be broadly categorized as
22
1. Point Anomalies
2. Contextual Anomalies
3. Collective Anomalies
Chandola, V., Banerjee, A. & Kumar, V., 2009. Anomaly Detection: A Survey. ACM Computing Surveys, July. 41(3).
If a collection of related data instances is
anomalous with respect to the entire data set, it
is termed as a collective anomaly

23. How to Identify Outliers in your Data
24
1. Extreme Value Analysis
2. Proximity Methods
3. Projection Methods
4. Methods Robust to Outliers
https://machinelearningmastery.com/how-to-identify-outliers-in-your-data/

24. How to Identify Outliers in your Data
25
1. Extreme Value Analysis
2. Proximity Methods
3. Projection Methods
4. Methods Robust to Outliers
https://machinelearningmastery.com/how-to-identify-outliers-in-your-data/
1. Focus on univariate methods
2. Visualize the data using scatterplots, histograms and box
and whisker plots and look for extreme values
3. Assume a distribution (Gaussian) and look for values more
than 2 or 3 standard deviations from the mean or 1.5 times
from the first or third quartile
4. Filter out outliers candidate from training dataset and
assess your models performance
https://en.wikipedia.org/wiki/Multivariate_normal_distribution

25. How to Identify Outliers in your Data
26
1. Extreme Value Analysis
2. Proximity Methods
3. Projection Methods
4. Methods Robust to Outliers
https://machinelearningmastery.com/how-to-identify-outliers-in-your-data/

26. How to Identify Outliers in your Data
27
1. Extreme Value Analysis
2. Proximity Methods
3. Projection Methods
4. Methods Robust to Outliers
https://machinelearningmastery.com/how-to-identify-outliers-in-your-data/

27. How to Identify Outliers in your Data
28
1. Extreme Value Analysis
2. Proximity Methods
3. Projection Methods
4. Methods Robust to Outliers
https://machinelearningmastery.com/how-to-identify-outliers-in-your-data/

28. Typology of Anomalies
29
https://tunguska.home.xs4all.nl/Publications/AD_Typology.htm

29. Anomaly based IDS using Backpropagation Neural Network
https://pdfs.semanticscholar.org/3ac9/37cb50bad238091fba6d1076a78136fe8691.pdf
• Vrushali D. Mane ME (Electronics) JNEC, Aurangabad
• S.N. Pawar Associate Professor JNEC, Aurangabad
• Neural Network detect 98% accuracy

30. Anomaly Detection for Timeseries

31. Recurrent neural network (RNN)

32. Recurrent neural network (GRU)
n=24

33. Recurrent neural network (GRU)

34. Historical

35. Historical

36. DB-Scan

37. Historical with DB-Scan
https://en.wikipedia.org/wiki/DBSCAN https://www.mathworks.com/matlabcentral/fileexchange/53842-dbscan

38. Historical with DB-Scan

39. RNN with DB-Scan

40. Historical with DB-Scan

41. Time Shift Detection

42. Create new graph with shift to 1 step

43. Zoom your graph and shift first 3 step

44. Zoom your graph and shift first 3 step

45. Zoom your graph and shift first 3 step

46. Zoom your graph and shift first 3 step

47. Cosine Similarity
https://www.softnix.co.th/2019/05/29/similarity-%E0%B8%84%E0%B8%A7%E0%B8%B2%E0%B8%A1%E0%B9%80%E0%B8%AB%E0

48. Rotate graph next 3 step and compare with cosine similarity
Remark when value lower than threshold

49. Compare with anomaly detection (left) and correlation values (right)

50. Pros & Cons
Time shift Detection DB-Scan
Work well on slope data like
temperature
Like Time shift compare
Not work on many spikes data Must defined eps and min_samples

52. Thank you