AUSOUG - Applied Machine Learning for Database Autonomous Health
•
1 gefällt mir•93 views
This session will focus on how Oracle is applying various Machine Learning technologies to the area of diagnostics to prevent performance issues and maintain availability. Details about the architectures and algorithms used for this Autonomous Health initiative will be covered for both Cloud and On-premise database deployments. Components and functionality covered will be Cluster Heath Advisor, Trace File Analyzer (Collector, Receiver and Web), Hang Manager, and Adaptive Bug Search
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie AUSOUG - Applied Machine Learning for Database Autonomous Health
Ähnlich wie AUSOUG - Applied Machine Learning for Database Autonomous Health (20)
Mehr von Sandesh Rao
Mehr von Sandesh Rao (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
AUSOUG - Applied Machine Learning for Database Autonomous Health
- 8. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Autonomous Health Cloud Platform MachinesSmart Collectors SRs Expert Input Feedback & Improvement SRs Model Generation Model Knowledge Extraction Applied Machine Learning Cloud Ops Object Store Admin UI in Control Plane Oracle Support Bug DB SE UI in Support Tenant (CNS) Cleansing, metadata creation & clustering 5 Model generation with expert scrubbing 6 Deployed as part of cloud image, running from the start 1 Proactive regular health checking, real- time fault detection, automatic incident analysis, diagnostic collection & masking of sensitive data 2 Use real-time health dashboards for anomaly detection, root cause analysis & push of proactive, preventative & corrective actions. Auto bug search & auto bug & SR creation. 3 Auto SR analysis, diagnosis assistance via automatic anomaly detection, collaboration and one click bug creation 4 Message Broker
- 9. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Why Applied Machine Learning? • Brings an application’s perspective versus a platform toolkit viewpoint • Brings data science, algorithms, and domain expertise together • Packages machine learning into usable, real-world operational algorithms and models that are applied at runtime • Produces results and recommendations easily understood and trusted by non-data scientist/analyst end-users
- 10. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Applied Machine Learning for Cloud Operations • Generic ML-extracted Data Clusters are insufficient for diagnostics • Operational data correlation does not determine root cause • Trusted root cause determination critical to swift corrective actions • Algorithms selected and models built require domain expertise • Models refined via field feedback Subject Matter Expert ASH ML Knowledge Extraction Model Generation Human Supervision Application Optimized Models Feedback Scrub Data
- 11. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Real-time Prevention • Data Ingestion – Kernel Smoothing and Moving Average – Interpolation and Imputation • Prediction and Pattern Recognition – Multivariate and Auto-Associative Regression – Clustering, Similarity Operators and Bayes Networks • Fault and Anomaly Detection – Sequential Probability Ratio Tests – Conditional Probability Filters & Hidden Markov Models • Prognosis and Diagnosis – Bayesian Belief Networks and Probabilistic Inference – Remaining Useful Life Regression and GPM Models Rapid Recovery Autonomous Health Platform ML Technologies • Data Ingestion – ELK – Lucene • Prediction and Pattern Recognition – TF-IDF and Bag-of-Words modelling – Sequence Matcher – K-nearest Neighbour • Fault and Anomaly Detection – Decision Trees and Random Forest – Sequential Pattern Mining • Prognosis and Diagnosis – Recurrent neural Network – Long short-term memory Predictive Analysis
- 27. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Trace File Analyzer – High Level Anomaly Detection Flow Log Cleansing 1 2 3 4 5 6 Entry Feature Creation Entry Clustering Model Generation Expert Input Knowledge Base Creation Knowledge Base Indexing Feedback Training Real-time Log File Processing Timestamp Correlation & Ranking 8 9 7 Batch Feedback
- 28. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Knowledge Base Indexing Entry Clustering Model Generation Entry Feature Creation Log Cleansing 1 2 3 4 5 6 Expert Input Knowledge Base Creation FeedbackTraining Real-time Log File Processing Timestamp Correlation & Ranking 8 97 Batch Feedback Log File Collection Data Cleansing & Reduction waited for 'ASM file metadata operation', seq_num: 29 2016-10-20 02:12:56.937 : OCRRAW:1: kgfo_kge2slos error stack at kgfoAl06: ORA-29701: unable to connect to Cluster Synchronization Service 2016-10-20 02:23:02.000 : OCRRAW:1: kgfo_kge2slos error stack at kgfoAl06: ORA-29701: unable to connect to Cluster Synchronization Service 2016-10-20 02:23:03.563 : OCRRAW:1: kgfo_kge2slos error stack at kgfoAl06: ORA-29701: unable to connect to Cluster Synchronization Service waited for [STR] seq_num: [NSTR] [NSTR] [NSTR] : [NSTR] [NSTR] unable to connect to Cluster Synchronization Service
- 29. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | .. Seen in Bugs Total Bugs Seen Seen in Files Total Files Seen Total Count .. .. 13 40 1440 5088 2890 .. Feature Extraction waited for [STR] seq_num: [NSTR] [NSTR] [NSTR] : [NSTR] [NSTR] unable to connect to Cluster Synchronization Service Knowledge Base Indexing Entry Clustering Model Generation Log Cleansing 1 3 4 5 6 Expert Input Knowledge Base Creation FeedbackTraining Real-time Log File Processing Timestamp Correlation & Ranking 8 97 Batch Feedback Entry Feature Creation 2
- 30. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Data Clustering Record Merging and feature aggregation for records belonging to same log signature Knowledge Base Indexing Entry Clustering Model Generation Log Cleansing 1 3 4 5 6 Expert Input Knowledge Base Creation FeedbackTraining Real-time Log File Processing Timestamp Correlation & Ranking 8 97 Batch Feedback 2 Entry Feature Creation
- 31. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Model Generation Data Clustering Expert Input Decision Tree Classifier First time labelling through functional rules Labelled dataset Result EvaluationUpdate Labelling 3 4 5 76 Entry Clustering Log Cleansing 1 3 Training Real-time Log File Processing Timestamp Correlation & Ranking 8 9 Batch Feedback 2 Knowledge Base Indexing Model Generation 4 5 6 Expert Input Knowledge Base Creation Feedback 7 Entry Feature Creation
- 32. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 32 Deploy with Minimum Footprint and Maximum Manageability Oracle 18c Domain Services Cluster Application Member Cluster Database Member Cluster Database Member Cluster Oracle Domain Services Cluster Database Member Cluster Application Member Cluster Database Member Cluster ORACLE CLUSTER DOMAIN Management Repository Service Trace File Analyzer Service Grid Names Service Storage Services QoS Management Service Rapid Home Provisioning Service Confidential – Oracle Internal/Restricted/Highly Restricted • Hosts Framework as Services • Reduces local resource footprint • Centralizes management • Speeds deployment and patching • Optional Shared Storage • Supports multiple versions and platforms going forward
- 33. 33
- 34. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Domain Services Cluster Already Has TFA User Interface • Central TFA Repository utilizing ACFS Storage • Member Clusters Send TFA Collections to the TFA Service on DSC • TFA Service indexes the Collection and runs Analysers. • New UI will be shipped in 19 34
- 35. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Standalone User Interface • TFA Collector will upload to central repository • TFA UI analyses files and generates – Events TimeLine – Anomaly TimeLine using Applied Machine Learning – Root Cause Analysis and Recommendations where available. – Interface to easily access all files and analyser reports. • Already used in Oracle Database Cloud. • Coming On Prem in 19 35
- 36. 36
- 37. 37
- 40. 40
- 44. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Database Health - Applied Machine Learning • Actual Internal and External customer data drives model development • Applied purpose-built Applied ML for knowledge extraction • Expert Dev team scrubs data • Generates Bayesian Network-based diagnostic root-cause models • Uses BN-based run-time models to perform real-time prognostics Discovers Potential Cluster & DB Problems CHA Dev Team ASH ML Knowledge Extraction BN Models Expert Supervision DB+Node Runtime Models Feedback Scrub Data CHA CHA
- 45. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Database Data Flow Overview Autonomous Health – Database Performance OS DataDB Data Database Prognostics Engine Alert & Preventive Action • Reads OS and DB Performance data directly from memory • Uses Machine Learning models and data to perform prognostics • Detects common RAC database problems • Performs root cause analysis • Sends alerts and preventative actions to Cloud Ops per target
- 46. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Data Sources and Data Points Autonomous Health – Database Performance Time CPU ASM IOPS Network % util Network_ Packets Dropped Log file sync Log file parallel write GC CR request GC current request GC current block 2-way GC current block busy Enq: CF - conten tion … 15:16:00 0.90 4100 13% 0 2 ms 600 us 0 0 300 us 1.5 ms 0 A Data Point contains > 150 signals (statistics and events) from multiple sources OS, ASM , Network DB ( SH, AWR session, system and PDB statistics ) Statistics are collected at a 1 second internal sampling rate , synchronized, smoothed and aggregated to a Data Point every 5 seconds
- 48. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Models Capture the Dynamic Behavior of all Normal Operation Models Capture all Normal Operating Modes 0 5000 10000 15000 20000 25000 30000 35000 40000 10:00 2:00 6:00 5100 9025 4024 2350 4100 22050 10000 21000 4400 2500 4900 800 IOPS user commits (/sec) log file parallel write (usec) log file sync (usec) A model captures the normal load phases and their statistics over time , and thus the characteristics for all load intensities and profiles . During monitoring , any data point similar to one of the vectors is NORMAL. One could say that the model REMEMBERS the normal operational dynamics over time In-Memory Reference Matrix (Part of “Normality” Model) IOPS #### 2500 4900 800 #### User Commits #### 10000 21000 4400 #### Log File Parallel Write #### 2350 4100 22050 #### Log File Sync #### 5100 9025 4024 #### … … … … … …
- 49. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | CHA Model: Find Similarity with Normal Values Autonomous Health – Database Performance Observed values (Part of a Data Point) CHA estimator/predictor (ESEE): “based on my normality model, the value of IOPS should be in the vicinity of ~ 4900, but it is reported as 10500, this is causing a residual of ~ 5600 in magnitude”, CHA fault detector: “such high magnitude of residuals should be tracked carefully! I’ll keep an eye on the incoming sequence of this signal IOPS and if it remains deviant I’ll generate a fault on it”. In-Memory Reference Matrix (Part of “Normality” Model) IOPS #### 2500 4900 800 #### User Commits #### 10000 21000 4400 #### Log File Parallel Write #### 2350 4100 22050 #### Log File Sync #### 5100 9025 4024 #### … … … … … … 10500 20000 4050 10250 … Residual Values (Part of a Data Point) 5600 -1000 -50 325 … Observed - Predicted =
- 50. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Inline and Immediate Fault Detection and Diagnostic Inference Autonomous Health – Database Performance Machine Learning, Pattern Recognition, & BN Engines Time CPU ASM IOPS Network % util Network_ Packets Dropped Log file sync Log file parallel write GC CR request GC current request GC current block 2-way GC current block busy Enq: CF - conten tion … 15:16:00 0.90 4100 88% 105 2 ms 600 us 504 ms 513 ms 2 ms 5.9 ms 0 15:16:00 OK OK HIGH 1 HIGH 2 OK OK HIGH 3 HIGH 3 HIGH 4 HIGH 4 OK Input : Data Point at Time t Fault Detection and Classification Diagnostic Inference 15:16:00 Symptoms 1. Network Bandwidth Utilization 2. Network Packet Loss 3. Global Cache Requests Incomplete 4. Global Cache Message Latency Root Cause (Target of Corrective Action) Network Bandwidth Utilization Diagnostic Inference Engine
- 52. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Calibrating CHA to your RAC deployment Confidential – Oracle Restricted 52 Choosing a Data Set for Calibration – Defining “normal” $ chactl query calibration –cluster –timeranges ‘start=2016-10-28 07:00:00,end=2016-10-28 13:00:00’ Cluster name : mycluster Start time : 2016-10-28 07:00:00 End time : 2016-10-28 13:00:00 Total Samples : 11524 Percentage of filtered data : 100% 1) Disk read (ASM) (Mbyte/sec) MEAN MEDIAN STDDEV MIN MAX 0.11 0.00 2.62 0.00 114.66 <25 <50 <75 <100 >=100 99.87% 0.08% 0.00% 0.02% 0.03% 2) Disk write (ASM) (Mbyte/sec) MEAN MEDIAN STDDEV MIN MAX 0.01 0.00 0.15 0.00 6.77 <50 <100 <150 <200 >=200 100.00% 0.00% 0.00% 0.00% 0.00% 3) Disk throughput (ASM) (IO/sec) MEAN MEDIAN STDDEV MIN MAX 2.20 0.00 31.17 0.00 1100.00 <5000 <10000 <15000 <20000 >=20000 100.00% 0.00% 0.00% 0.00% 0.00% 4) CPU utilization (total) (%) MEAN MEDIAN STDDEV MIN MAX 9.62 9.30 7.95 1.80 77.90 <20 <40 <60 <80 >=80 92.67% 6.17% 1.11% 0.05% 0.00%
- 53. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Calibrating CHA to your RAC deployment • Create and store the new model $ chactl query calibrate cluster –model daytime –timeranges ‘start=2018-10-28 07:00:00, end=2018-10-28 13:00:00’ • Begin using the new model $ chactl monitor cluster –model daytime • Confirm the new model is being used $ chactl status –verbose monitoring nodes svr01, svr02 using model daytime monitoring database qoltpacdb, instances oltpacdb_1, oltpacdb_2 using model DEFAULT_DB Confidential – Oracle Restricted 53 Creating a new CHA Model with CHACTL
- 54. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Cluster Health Advisor – Command Line Operations Confidential – Oracle Restricted 54 Monitoring Your Databases and Nodes with CHACTL Enable CHA monitoring on RAC database with optional model $ chactl monitor database –db oltpacdb [-model model_name] Enable CHA monitoring on RAC database with optional verbose $ chactl status –verbose monitoring nodes svr01, svr02 using model DEFAULT_CLUSTER monitoring database oltpacdb, instances oltpacdb_1, oltpacdb_2 using model DEFAULT_DB
- 55. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | CHA Command Line Operations Confidential – Oracle Restricted 55 Checking for Health Issues and Corrective Actions with CHACTL QUERY DIAGNOSIS $ chactl query diagnosis -db oltpacdb -start "2016-10-28 01:52:50" -end "2016-10-28 03:19:15" 2016-10-28 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_1) [detected] 2016-10-28 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_2) [detected] 2016-10-28 02:59:35.0 Database oltpacdb DB Log File Switch (oltpacdb_1) [detected] 2016-10-28 02:59:45.0 Database oltpacdb DB Log File Switch (oltpacdb_2) [detected] Problem: DB Control File IO Performance Description: CHA has detected that reads or writes to the control files are slower than expected. Cause: The Cluster Health Advisor (CHA) detected that reads or writes to the control files were slow because of an increase in disk IO. The slow control file reads and writes may have an impact on checkpoint and Log Writer (LGWR) performance. Action: Separate the control files from other database files and move them to faster disks or Solid State Devices. Problem: DB Log File Switch Description: CHA detected that database sessions are waiting longer than expected for log switch completions. Cause: The Cluster Health Advisor (CHA) detected high contention during log switches because the redo log files were small and the redo logs switched frequently. Action: Increase the size of the redo logs.
- 56. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Cluster Health Advisor – Command Line Operations Confidential – Oracle Restricted 56 HTML Diagnostic Health Output Available (-html <file_name>)