SlideShare ist ein Scribd-Unternehmen logo

Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elastic Cloud Enterprise at Otto

•
•
Elasticsearch
Elasticsearch

See how the Otto.de team built a scalable and resilient logging solution and how they’re scaling Logstash, addressing housekeeping for Elasticsearch, and collecting usage metrics for analytics and billing.

Technologie
1 von 30
Christian Herweg Thomas Klinger Lukas Kucharski DIVIDE & CONQUER Logging Architecture in Distributed Ecosystems with Elastic Cloud Enterprise
Elastic{ON} Tour: Frankfurt 2018 Facts & Figures 07.11.18 2 • Location: 
 Hamburg (head office) • Employees: 
 4,700 (all locations) • Sales 2016/2017: 
 2.9 billion euros OTTO campus in Hamburg
Elastic{ON} Tour: Frankfurt 2018 OTTO – Number 1 in Fashion & Lifestyle* 07.11.18 3 *B2C mail order, GfK figures 2014 6.800
 brands (in-house and many premium third-party brands) Over 2,8 million items online Huge product portfolio from fashion and lifestyle to household appliances and multimedia, DIY, kitchens, furniture and toys 120 
 specialist catalogues The only ‘big book’ company to make the jump into the digital world 6 specialist online shops
Elastic{ON} Tour: Frankfurt 2018 Business Domains Mirror the System Architecture Page Assembly Tesla ShopOffice AfterSales Search P13N Order User Authentication Tracking u.v.m. Code ownership results in high code quality ! Continuous Delivery 
 permits more than 800 deployments per week. ! Verticals develop and test features fast and independently ! schematic presentation Business phases of the order process frame the distributed and parallel development. The technical mirror to this phase model allows highest possible flexibility in business concept development. ! 07.11.18 4
Elastic{ON} Tour: Frankfurt 2018 07.11.18 5 Step by Step Fragmentation of otto.de Dedicated Monolith Vertical Architecture Micro Services & „Cloud Readiness“ I n d e xI n d e x I n d e x I n d e x I n d e x I n d e x Serverless & Cloud 2011 df /var/log 2013
 ~3TB data 2015
 ~17TB data 2018
 up to 42TB data &
 splunk >
Elastic{ON} Tour: Frankfurt 2018 DISTRIBUTED LOGGING @ AWS a.k.a. „How to enable tenants to log data“ 07.11.18 6
Elastic{ON} Tour: Frankfurt 2018 Requirements • Security • Encryption (at rest, in transit) • Authentication & Authorization • Isolation of resources („multi-tenancy“) • Accessing other vertical‘s logs => Rethink classic operations model, become a service provider 07.11.18 7
Elastic{ON} Tour: Frankfurt 2018 Core Principles • Multi tenancy • Shared responsibility • Security by design • Automation (Goal: Provision the logging platform during lunch break) 07.11.18 8
Elastic{ON} Tour: Frankfurt 2018 Elasticsearch Deployment at Scale • AWS Elasticsearch (Cost, missing encryption) • Elasticsearch on EC2 (Technical overhead) • Elastic Cloud Enterprise (fits requirements) 07.11.18 9
Elastic{ON} Tour: Frankfurt 2018 Challenges • AWS Cross Account actions • (Near-)Realtime processing of logs • Processing multiple data formats (JSON, Syslog ...) • Queueing input data for failure scenarios • Autoscaling Logstash • Automation of Elasticsearch cluster management • Keeping up with new features in the Elastic stack 07.11.18 10
Elastic{ON} Tour: Frankfurt 2018 ELASTIC CLOUD ENTERPRISE a.k.a. „How to provision tons of Elasticsearch and Kibana clusters“ 07.11.18 11
Elastic{ON} Tour: Frankfurt 2018 How Elastic Cloud Enterprise Helped Us • No need to build custom provisioning service for Elasticsearch and Kibana clusters • Provides security features via Elastic features (authentication, authorization, integration with LDAP) • Supports multiple Elastic stack versions • Easy to set up a basic installation • Customizable (stack packs, underlying EC2 instance) • Extensive API • Updates do not require downtime • Multi-tenancy 07.11.18 12
Elastic{ON} Tour: Frankfurt 2018 Architecture 07.11.18 13 Key facts: • 2 Loadbalancers • 3 ECE Proxies • 3 ECE Coordinators • 21 ECE Allocators (i3.4xlarge) • 20+ Clusters • Dedicated Monitoring Cluster
Elastic{ON} Tour: Frankfurt 2018 Challenges • Automated deployment • Stateful components in a cloud environment (allocators) • Troubleshooting ECE • No fine-grained authentication and authorization (LDAP, OAuth2) except readonly/root users => Custom tooling required 07.11.18 14
Elastic{ON} Tour: Frankfurt 2018 LOGINGEST a.k.a. „How to move logs from one account to another“ 07.11.18 15
Elastic{ON} Tour: Frankfurt 2018 Challenges • Cross account log ingestion • Pipeline performance & autoscaling • Finding a suitable input • S3, CloudWatch Logs (Logstash state management) • Kinesis, Kafka (too expensive) • S3 & SNS & SQS (logstash-input-s3-sns-sqs) • Pipeline management (in Kibana) 07.11.18 16
Elastic{ON} Tour: Frankfurt 2018 Architecture Logingest 07.11.18 17
Elastic{ON} Tour: Frankfurt 2018 Autoscaling logstash 07.11.18 18 • Autoscale between 30 and 110 Logstash containers in total • Scaling based on demand and cluster size • Manual intervention possible • consume unexpected peaks • stopping ingest in failure scenarios
Elastic{ON} Tour: Frankfurt 2018 HOUSEKEEPING a.k.a. „How to keep your clusters clean“ 07.11.18 19
Elastic{ON} Tour: Frankfurt 2018 Curator as a Service • Verticals provide configuration in Git • Synchronized to S3 • Master-worker architecture based on AWS Lambda, CloudWatch and SQS 07.11.18 20
Elastic{ON} Tour: Frankfurt 2018 Challenges • Verticals are responsible for curator configuration • Lack of knowledge (both Curator and Elasticsearch) • Monitoring Curator • Tracking origins of failed runs (misconfiguration, internal failure) • Transparency for verticals • AWS Lambda limits (max. 3 minutes runtime) 07.11.18 21
Elastic{ON} Tour: Frankfurt 2018 LESSONS LEARNED a.k.a. „Learning through failure“ 07.11.18 22
Elastic{ON} Tour: Frankfurt 2018 Learnings • Spread knowledge about Elasticsearch with teams • Create a sustainable knowledgebase • Automation is essential • Reduce operational overhead • Have time to develop and introduce new features • Know your I/O limits and requirements • Scaling Logstash is not a trivial task 07.11.18 23
Elastic{ON} Tour: Frankfurt 2018 BACKLOG a.k.a. „Still some work to do“ 07.11.18 24
Elastic{ON} Tour: Frankfurt 2018 Next steps • Develop custom Logstash pipeline management solution • Move Housekeeping workers from AWS Lambda to AWS Fargate • Evaluate Index Lifecyle Management via Elasticsearch • Upgrade to ECE 2.0 • Leverage potential of ECE & Elastic features • Tenants use and know about Machine Learning & APM • ECE 2.0 features • When available: cross-cluster search 07.11.18 25
Elastic{ON} Tour: Frankfurt 2018 WE ARE HIRING! https://www.otto.de/jobs 07.11.18 26
Elastic{ON} Tour: Frankfurt 2018 THANK YOU FOR YOUR ATTENTION 07.11.18 27
Elastic{ON} Tour: Frankfurt 2018 BACKUP 07.11.18 28
Elastic{ON} Tour: Frankfurt 2018 History – The Beginning... 07.11.18 29
K2 - Cloud Readiness 08.05.2018 30 Autonomous and team specific utilization of technology. Decoupling of interfaces and system components reduces architecture complexity and interdependences especially with service components. Perimeter protection is replaced by an integrated security concept, reducing vulnerability and allowing customised implementations. Cloud migration allows new sourcing and scaling models

Empfohlen

Security Events Logging at Bell with the Elastic Stack
Security Events Logging at Bell with the Elastic StackSecurity Events Logging at Bell with the Elastic Stack
Security Events Logging at Bell with the Elastic StackElasticsearch
 
Artik cloud deview 2016
Artik cloud deview 2016Artik cloud deview 2016
Artik cloud deview 2016NAVER D2
 
Turning Evidence into Insights: How NCIS Leverages Elastic
Turning Evidence into Insights: How NCIS Leverages Elastic Turning Evidence into Insights: How NCIS Leverages Elastic
Turning Evidence into Insights: How NCIS Leverages Elastic Elasticsearch
 
Concept to reality: An advanced agile integration blueprint
Concept to reality: An advanced agile integration blueprintConcept to reality: An advanced agile integration blueprint
Concept to reality: An advanced agile integration blueprintEric D. Schabell
 
Empower Your Security Practitioners with Elastic SIEM
Empower Your Security Practitioners with Elastic SIEMEmpower Your Security Practitioners with Elastic SIEM
Empower Your Security Practitioners with Elastic SIEMElasticsearch
 
Siscale Lightning Talk: Automated Root Cause Analysis with Elastic Stack
Siscale Lightning Talk: Automated Root Cause Analysis with Elastic StackSiscale Lightning Talk: Automated Root Cause Analysis with Elastic Stack
Siscale Lightning Talk: Automated Root Cause Analysis with Elastic StackElasticsearch
 
Building a reliable and cost effect logging system at Box
Building a reliable and cost effect logging system at Box Building a reliable and cost effect logging system at Box
Building a reliable and cost effect logging system at Box Elasticsearch
 
How KeyBank Used Elastic to Build an Enterprise Monitoring Solution
How KeyBank Used Elastic to Build an Enterprise Monitoring SolutionHow KeyBank Used Elastic to Build an Enterprise Monitoring Solution
How KeyBank Used Elastic to Build an Enterprise Monitoring SolutionElasticsearch
 

Empfohlen

Security Events Logging at Bell with the Elastic Stack
Security Events Logging at Bell with the Elastic StackSecurity Events Logging at Bell with the Elastic Stack
Security Events Logging at Bell with the Elastic StackElasticsearch
 
Artik cloud deview 2016
Artik cloud deview 2016Artik cloud deview 2016
Artik cloud deview 2016NAVER D2
 
Turning Evidence into Insights: How NCIS Leverages Elastic
Turning Evidence into Insights: How NCIS Leverages Elastic Turning Evidence into Insights: How NCIS Leverages Elastic
Turning Evidence into Insights: How NCIS Leverages Elastic Elasticsearch
 
Concept to reality: An advanced agile integration blueprint
Concept to reality: An advanced agile integration blueprintConcept to reality: An advanced agile integration blueprint
Concept to reality: An advanced agile integration blueprintEric D. Schabell
 
Empower Your Security Practitioners with Elastic SIEM
Empower Your Security Practitioners with Elastic SIEMEmpower Your Security Practitioners with Elastic SIEM
Empower Your Security Practitioners with Elastic SIEMElasticsearch
 
Siscale Lightning Talk: Automated Root Cause Analysis with Elastic Stack
Siscale Lightning Talk: Automated Root Cause Analysis with Elastic StackSiscale Lightning Talk: Automated Root Cause Analysis with Elastic Stack
Siscale Lightning Talk: Automated Root Cause Analysis with Elastic StackElasticsearch
 
Building a reliable and cost effect logging system at Box
Building a reliable and cost effect logging system at Box Building a reliable and cost effect logging system at Box
Building a reliable and cost effect logging system at Box Elasticsearch
 
How KeyBank Used Elastic to Build an Enterprise Monitoring Solution
How KeyBank Used Elastic to Build an Enterprise Monitoring SolutionHow KeyBank Used Elastic to Build an Enterprise Monitoring Solution
How KeyBank Used Elastic to Build an Enterprise Monitoring SolutionElasticsearch
 
Alex Nauda [Nobl9] | How Not to Build an SLO Platform | InfluxDays NA 2021
Alex Nauda [Nobl9] | How Not to Build an SLO Platform | InfluxDays NA 2021Alex Nauda [Nobl9] | How Not to Build an SLO Platform | InfluxDays NA 2021
Alex Nauda [Nobl9] | How Not to Build an SLO Platform | InfluxDays NA 2021InfluxData
 
Kafka and Kafka Streams in the Global Schibsted Data Platform
Kafka and Kafka Streams in the Global Schibsted Data PlatformKafka and Kafka Streams in the Global Schibsted Data Platform
Kafka and Kafka Streams in the Global Schibsted Data PlatformFredrik Vraalsen
 
Elastic on a Hyper-Converged Infrastructure for Operational Log Analytics
Elastic on a Hyper-Converged Infrastructure for Operational Log AnalyticsElastic on a Hyper-Converged Infrastructure for Operational Log Analytics
Elastic on a Hyper-Converged Infrastructure for Operational Log AnalyticsElasticsearch
 
What’s Evolving in the Elastic Stack
What’s Evolving in the Elastic StackWhat’s Evolving in the Elastic Stack
What’s Evolving in the Elastic StackElasticsearch
 
Monitoring docker, k8s and your applications with the elastic stack
Monitoring docker, k8s and your applications with the elastic stackMonitoring docker, k8s and your applications with the elastic stack
Monitoring docker, k8s and your applications with the elastic stackSmartWave
 
[WSO2Con USA 2018] Microservices, Containers, and Beyond
[WSO2Con USA 2018] Microservices, Containers, and Beyond[WSO2Con USA 2018] Microservices, Containers, and Beyond
[WSO2Con USA 2018] Microservices, Containers, and BeyondWSO2
 
Hunting for Evil with the Elastic Stack
Hunting for Evil with the Elastic StackHunting for Evil with the Elastic Stack
Hunting for Evil with the Elastic StackElasticsearch
 
How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...
How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...
How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...InfluxData
 
Elastic @ John Deere
Elastic @ John DeereElastic @ John Deere
Elastic @ John DeereElasticsearch
 
SnapLogic Live: IoT Integration
SnapLogic Live: IoT IntegrationSnapLogic Live: IoT Integration
SnapLogic Live: IoT IntegrationSnapLogic
 
Better Search and Business Analytics at Southern Glazer’s Wine & Spirits
Better Search and Business Analytics at Southern Glazer’s Wine & SpiritsBetter Search and Business Analytics at Southern Glazer’s Wine & Spirits
Better Search and Business Analytics at Southern Glazer’s Wine & SpiritsElasticsearch
 
Elastic at KPN
Elastic at KPNElastic at KPN
Elastic at KPNElasticsearch
 
Scalable Application Development @ Picnic
Scalable Application Development @ PicnicScalable Application Development @ Picnic
Scalable Application Development @ PicnicSander Mak (@Sander_Mak)
 
Infrastructure monitoring made easy, from ingest to insight
Infrastructure monitoring made easy, from ingest to insightInfrastructure monitoring made easy, from ingest to insight
Infrastructure monitoring made easy, from ingest to insightElasticsearch
 
Achieving cyber mission assurance with near real-time impact
Achieving cyber mission assurance with near real-time impactAchieving cyber mission assurance with near real-time impact
Achieving cyber mission assurance with near real-time impactElasticsearch
 
Log Monitoring and Anomaly Detection at Scale at ORNL
Log Monitoring and Anomaly Detection at Scale at ORNLLog Monitoring and Anomaly Detection at Scale at ORNL
Log Monitoring and Anomaly Detection at Scale at ORNLElasticsearch
 
CSX: Real-time Business Discovery with the Elastic Stack
CSX: Real-time Business Discovery with the Elastic StackCSX: Real-time Business Discovery with the Elastic Stack
CSX: Real-time Business Discovery with the Elastic StackElasticsearch
 
Elasticsearch on Azure
Elasticsearch on AzureElasticsearch on Azure
Elasticsearch on AzureElasticsearch
 
Improving search at Wellcome Collection
Improving search at Wellcome CollectionImproving search at Wellcome Collection
Improving search at Wellcome CollectionElasticsearch
 
Machine Learning for Anomaly Detection, Time Series Modeling, and More
Machine Learning for Anomaly Detection, Time Series Modeling, and MoreMachine Learning for Anomaly Detection, Time Series Modeling, and More
Machine Learning for Anomaly Detection, Time Series Modeling, and MoreElasticsearch
 
Elastic Meetup Belgium - December 2018
Elastic Meetup Belgium - December 2018Elastic Meetup Belgium - December 2018
Elastic Meetup Belgium - December 2018Arthur Eyckerman
 
Updates from Hungary (Jozsef Kovacs)
Updates from Hungary (Jozsef Kovacs)Updates from Hungary (Jozsef Kovacs)
Updates from Hungary (Jozsef Kovacs)EOSC-hub project
 

Weitere ähnliche Inhalte

Was ist angesagt?

Alex Nauda [Nobl9] | How Not to Build an SLO Platform | InfluxDays NA 2021
Alex Nauda [Nobl9] | How Not to Build an SLO Platform | InfluxDays NA 2021Alex Nauda [Nobl9] | How Not to Build an SLO Platform | InfluxDays NA 2021
Alex Nauda [Nobl9] | How Not to Build an SLO Platform | InfluxDays NA 2021InfluxData
 
Kafka and Kafka Streams in the Global Schibsted Data Platform
Kafka and Kafka Streams in the Global Schibsted Data PlatformKafka and Kafka Streams in the Global Schibsted Data Platform
Kafka and Kafka Streams in the Global Schibsted Data PlatformFredrik Vraalsen
 
Elastic on a Hyper-Converged Infrastructure for Operational Log Analytics
Elastic on a Hyper-Converged Infrastructure for Operational Log AnalyticsElastic on a Hyper-Converged Infrastructure for Operational Log Analytics
Elastic on a Hyper-Converged Infrastructure for Operational Log AnalyticsElasticsearch
 
What’s Evolving in the Elastic Stack
What’s Evolving in the Elastic StackWhat’s Evolving in the Elastic Stack
What’s Evolving in the Elastic StackElasticsearch
 
Monitoring docker, k8s and your applications with the elastic stack
Monitoring docker, k8s and your applications with the elastic stackMonitoring docker, k8s and your applications with the elastic stack
Monitoring docker, k8s and your applications with the elastic stackSmartWave
 
[WSO2Con USA 2018] Microservices, Containers, and Beyond
[WSO2Con USA 2018] Microservices, Containers, and Beyond[WSO2Con USA 2018] Microservices, Containers, and Beyond
[WSO2Con USA 2018] Microservices, Containers, and BeyondWSO2
 
Hunting for Evil with the Elastic Stack
Hunting for Evil with the Elastic StackHunting for Evil with the Elastic Stack
Hunting for Evil with the Elastic StackElasticsearch
 
How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...
How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...
How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...InfluxData
 
Elastic @ John Deere
Elastic @ John DeereElastic @ John Deere
Elastic @ John DeereElasticsearch
 
SnapLogic Live: IoT Integration
SnapLogic Live: IoT IntegrationSnapLogic Live: IoT Integration
SnapLogic Live: IoT IntegrationSnapLogic
 
Better Search and Business Analytics at Southern Glazer’s Wine & Spirits
Better Search and Business Analytics at Southern Glazer’s Wine & SpiritsBetter Search and Business Analytics at Southern Glazer’s Wine & Spirits
Better Search and Business Analytics at Southern Glazer’s Wine & SpiritsElasticsearch
 
Infrastructure monitoring made easy, from ingest to insight
Infrastructure monitoring made easy, from ingest to insightInfrastructure monitoring made easy, from ingest to insight
Infrastructure monitoring made easy, from ingest to insightElasticsearch
 
Achieving cyber mission assurance with near real-time impact
Achieving cyber mission assurance with near real-time impactAchieving cyber mission assurance with near real-time impact
Achieving cyber mission assurance with near real-time impactElasticsearch
 
Log Monitoring and Anomaly Detection at Scale at ORNL
Log Monitoring and Anomaly Detection at Scale at ORNLLog Monitoring and Anomaly Detection at Scale at ORNL
Log Monitoring and Anomaly Detection at Scale at ORNLElasticsearch
 
CSX: Real-time Business Discovery with the Elastic Stack
CSX: Real-time Business Discovery with the Elastic StackCSX: Real-time Business Discovery with the Elastic Stack
CSX: Real-time Business Discovery with the Elastic StackElasticsearch
 
Elasticsearch on Azure
Elasticsearch on AzureElasticsearch on Azure
Elasticsearch on AzureElasticsearch
 
Improving search at Wellcome Collection
Improving search at Wellcome CollectionImproving search at Wellcome Collection
Improving search at Wellcome CollectionElasticsearch
 
Machine Learning for Anomaly Detection, Time Series Modeling, and More
Machine Learning for Anomaly Detection, Time Series Modeling, and MoreMachine Learning for Anomaly Detection, Time Series Modeling, and More
Machine Learning for Anomaly Detection, Time Series Modeling, and MoreElasticsearch
 

Was ist angesagt? (20)

Alex Nauda [Nobl9] | How Not to Build an SLO Platform | InfluxDays NA 2021
Alex Nauda [Nobl9] | How Not to Build an SLO Platform | InfluxDays NA 2021Alex Nauda [Nobl9] | How Not to Build an SLO Platform | InfluxDays NA 2021
Alex Nauda [Nobl9] | How Not to Build an SLO Platform | InfluxDays NA 2021
 
Kafka and Kafka Streams in the Global Schibsted Data Platform
Kafka and Kafka Streams in the Global Schibsted Data PlatformKafka and Kafka Streams in the Global Schibsted Data Platform
Kafka and Kafka Streams in the Global Schibsted Data Platform
 
Elastic on a Hyper-Converged Infrastructure for Operational Log Analytics
Elastic on a Hyper-Converged Infrastructure for Operational Log AnalyticsElastic on a Hyper-Converged Infrastructure for Operational Log Analytics
Elastic on a Hyper-Converged Infrastructure for Operational Log Analytics
 
What’s Evolving in the Elastic Stack
What’s Evolving in the Elastic StackWhat’s Evolving in the Elastic Stack
What’s Evolving in the Elastic Stack
 
Monitoring docker, k8s and your applications with the elastic stack
Monitoring docker, k8s and your applications with the elastic stackMonitoring docker, k8s and your applications with the elastic stack
Monitoring docker, k8s and your applications with the elastic stack
 
[WSO2Con USA 2018] Microservices, Containers, and Beyond
[WSO2Con USA 2018] Microservices, Containers, and Beyond[WSO2Con USA 2018] Microservices, Containers, and Beyond
[WSO2Con USA 2018] Microservices, Containers, and Beyond
 
Hunting for Evil with the Elastic Stack
Hunting for Evil with the Elastic StackHunting for Evil with the Elastic Stack
Hunting for Evil with the Elastic Stack
 
How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...
How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...
How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...
 
Elastic @ John Deere
Elastic @ John DeereElastic @ John Deere
Elastic @ John Deere
 
SnapLogic Live: IoT Integration
SnapLogic Live: IoT IntegrationSnapLogic Live: IoT Integration
SnapLogic Live: IoT Integration
 
Better Search and Business Analytics at Southern Glazer’s Wine & Spirits
Better Search and Business Analytics at Southern Glazer’s Wine & SpiritsBetter Search and Business Analytics at Southern Glazer’s Wine & Spirits
Better Search and Business Analytics at Southern Glazer’s Wine & Spirits
 
Elastic at KPN
Elastic at KPNElastic at KPN
Elastic at KPN
 
Scalable Application Development @ Picnic
Scalable Application Development @ PicnicScalable Application Development @ Picnic
Scalable Application Development @ Picnic
 
Infrastructure monitoring made easy, from ingest to insight
Infrastructure monitoring made easy, from ingest to insightInfrastructure monitoring made easy, from ingest to insight
Infrastructure monitoring made easy, from ingest to insight
 
Achieving cyber mission assurance with near real-time impact
Achieving cyber mission assurance with near real-time impactAchieving cyber mission assurance with near real-time impact
Achieving cyber mission assurance with near real-time impact
 
Log Monitoring and Anomaly Detection at Scale at ORNL
Log Monitoring and Anomaly Detection at Scale at ORNLLog Monitoring and Anomaly Detection at Scale at ORNL
Log Monitoring and Anomaly Detection at Scale at ORNL
 
CSX: Real-time Business Discovery with the Elastic Stack
CSX: Real-time Business Discovery with the Elastic StackCSX: Real-time Business Discovery with the Elastic Stack
CSX: Real-time Business Discovery with the Elastic Stack
 
Elasticsearch on Azure
Elasticsearch on AzureElasticsearch on Azure
Elasticsearch on Azure
 
Improving search at Wellcome Collection
Improving search at Wellcome CollectionImproving search at Wellcome Collection
Improving search at Wellcome Collection
 
Machine Learning for Anomaly Detection, Time Series Modeling, and More
Machine Learning for Anomaly Detection, Time Series Modeling, and MoreMachine Learning for Anomaly Detection, Time Series Modeling, and More
Machine Learning for Anomaly Detection, Time Series Modeling, and More
 

Ähnlich wie Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elastic Cloud Enterprise at Otto

Elastic Meetup Belgium - December 2018
Elastic Meetup Belgium - December 2018Elastic Meetup Belgium - December 2018
Elastic Meetup Belgium - December 2018Arthur Eyckerman
 
Updates from Hungary (Jozsef Kovacs)
Updates from Hungary (Jozsef Kovacs)Updates from Hungary (Jozsef Kovacs)
Updates from Hungary (Jozsef Kovacs)EOSC-hub project
 
IoT interoperability
IoT interoperabilityIoT interoperability
IoT interoperability1248 Ltd.
 
KubeCon EU 2016: SmartCity IoT on Kubernetes
KubeCon EU 2016: SmartCity IoT on KubernetesKubeCon EU 2016: SmartCity IoT on Kubernetes
KubeCon EU 2016: SmartCity IoT on KubernetesKubeAcademy
 
SmartCity IoT on Kubernetes and OpenStack
SmartCity IoT on Kubernetes and OpenStackSmartCity IoT on Kubernetes and OpenStack
SmartCity IoT on Kubernetes and OpenStackJakub Pavlik
 
From BI Developer to Data Engineer with Oracle Analytics Cloud Data Lake Edition
From BI Developer to Data Engineer with Oracle Analytics Cloud Data Lake EditionFrom BI Developer to Data Engineer with Oracle Analytics Cloud Data Lake Edition
From BI Developer to Data Engineer with Oracle Analytics Cloud Data Lake EditionRittman Analytics
 
Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with ...
Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with ...Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with ...
Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with ...Miguel Pérez Colino
 
Automate your Oracle Cloud Infrastructure operation
Automate your Oracle Cloud Infrastructure operationAutomate your Oracle Cloud Infrastructure operation
Automate your Oracle Cloud Infrastructure operationNelson Calero
 
Bilot Azure on SAP Breakfast Club 16.05.2018
Bilot Azure on SAP Breakfast Club 16.05.2018Bilot Azure on SAP Breakfast Club 16.05.2018
Bilot Azure on SAP Breakfast Club 16.05.2018Bilot
 
How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.Renzo Tomà
 
ClickHouse Paris Meetup. ClickHouse Analytical DBMS, Introduction. By Alexand...
ClickHouse Paris Meetup. ClickHouse Analytical DBMS, Introduction. By Alexand...ClickHouse Paris Meetup. ClickHouse Analytical DBMS, Introduction. By Alexand...
ClickHouse Paris Meetup. ClickHouse Analytical DBMS, Introduction. By Alexand...Altinity Ltd
 
Combinação de logs, métricas e rastreamentos para observabilidade unificada
Combinação de logs, métricas e rastreamentos para observabilidade unificadaCombinação de logs, métricas e rastreamentos para observabilidade unificada
Combinação de logs, métricas e rastreamentos para observabilidade unificadaElasticsearch
 
Ovh analytics data compute with apache spark as a service meetup ovh bordeaux
Ovh analytics data compute with apache spark as a service meetup ovh bordeauxOvh analytics data compute with apache spark as a service meetup ovh bordeaux
Ovh analytics data compute with apache spark as a service meetup ovh bordeauxMojtaba Imani
 
OVH Analytics Data Compute - Apache Spark Cluster as a Service
OVH Analytics Data Compute - Apache Spark Cluster as a ServiceOVH Analytics Data Compute - Apache Spark Cluster as a Service
OVH Analytics Data Compute - Apache Spark Cluster as a ServiceOVHcloud
 
Google Cloud and Data Pipeline Patterns
Google Cloud and Data Pipeline PatternsGoogle Cloud and Data Pipeline Patterns
Google Cloud and Data Pipeline PatternsLynn Langit
 
Uri Budnik Stories From The Trenches
Uri Budnik Stories From The TrenchesUri Budnik Stories From The Trenches
Uri Budnik Stories From The TrenchesCloudcamp
 
Bandwidth: Use Cases for Elastic Cloud on Kubernetes
Bandwidth: Use Cases for Elastic Cloud on Kubernetes Bandwidth: Use Cases for Elastic Cloud on Kubernetes
Bandwidth: Use Cases for Elastic Cloud on Kubernetes Elasticsearch
 

Ähnlich wie Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elastic Cloud Enterprise at Otto (20)

Elastic Meetup Belgium - December 2018
Elastic Meetup Belgium - December 2018Elastic Meetup Belgium - December 2018
Elastic Meetup Belgium - December 2018
 
Updates from Hungary (Jozsef Kovacs)
Updates from Hungary (Jozsef Kovacs)Updates from Hungary (Jozsef Kovacs)
Updates from Hungary (Jozsef Kovacs)
 
IoT interoperability
IoT interoperabilityIoT interoperability
IoT interoperability
 
KubeCon EU 2016: SmartCity IoT on Kubernetes
KubeCon EU 2016: SmartCity IoT on KubernetesKubeCon EU 2016: SmartCity IoT on Kubernetes
KubeCon EU 2016: SmartCity IoT on Kubernetes
 
SmartCity IoT on Kubernetes and OpenStack
SmartCity IoT on Kubernetes and OpenStackSmartCity IoT on Kubernetes and OpenStack
SmartCity IoT on Kubernetes and OpenStack
 
From BI Developer to Data Engineer with Oracle Analytics Cloud Data Lake Edition
From BI Developer to Data Engineer with Oracle Analytics Cloud Data Lake EditionFrom BI Developer to Data Engineer with Oracle Analytics Cloud Data Lake Edition
From BI Developer to Data Engineer with Oracle Analytics Cloud Data Lake Edition
 
Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with ...
Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with ...Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with ...
Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with ...
 
Automate your Oracle Cloud Infrastructure operation
Automate your Oracle Cloud Infrastructure operationAutomate your Oracle Cloud Infrastructure operation
Automate your Oracle Cloud Infrastructure operation
 
AWS in Practice
AWS in PracticeAWS in Practice
AWS in Practice
 
DevOpsDays Amsterdam 2016 workshop
DevOpsDays Amsterdam 2016 workshopDevOpsDays Amsterdam 2016 workshop
DevOpsDays Amsterdam 2016 workshop
 
Bilot Azure on SAP Breakfast Club 16.05.2018
Bilot Azure on SAP Breakfast Club 16.05.2018Bilot Azure on SAP Breakfast Club 16.05.2018
Bilot Azure on SAP Breakfast Club 16.05.2018
 
How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.
 
ClickHouse Paris Meetup. ClickHouse Analytical DBMS, Introduction. By Alexand...
ClickHouse Paris Meetup. ClickHouse Analytical DBMS, Introduction. By Alexand...ClickHouse Paris Meetup. ClickHouse Analytical DBMS, Introduction. By Alexand...
ClickHouse Paris Meetup. ClickHouse Analytical DBMS, Introduction. By Alexand...
 
Flink Meetup Septmeber 2017 2018
Flink Meetup Septmeber 2017 2018Flink Meetup Septmeber 2017 2018
Flink Meetup Septmeber 2017 2018
 
Combinação de logs, métricas e rastreamentos para observabilidade unificada
Combinação de logs, métricas e rastreamentos para observabilidade unificadaCombinação de logs, métricas e rastreamentos para observabilidade unificada
Combinação de logs, métricas e rastreamentos para observabilidade unificada
 
Ovh analytics data compute with apache spark as a service meetup ovh bordeaux
Ovh analytics data compute with apache spark as a service meetup ovh bordeauxOvh analytics data compute with apache spark as a service meetup ovh bordeaux
Ovh analytics data compute with apache spark as a service meetup ovh bordeaux
 
OVH Analytics Data Compute - Apache Spark Cluster as a Service
OVH Analytics Data Compute - Apache Spark Cluster as a ServiceOVH Analytics Data Compute - Apache Spark Cluster as a Service
OVH Analytics Data Compute - Apache Spark Cluster as a Service
 
Google Cloud and Data Pipeline Patterns
Google Cloud and Data Pipeline PatternsGoogle Cloud and Data Pipeline Patterns
Google Cloud and Data Pipeline Patterns
 
Uri Budnik Stories From The Trenches
Uri Budnik Stories From The TrenchesUri Budnik Stories From The Trenches
Uri Budnik Stories From The Trenches
 
Bandwidth: Use Cases for Elastic Cloud on Kubernetes
Bandwidth: Use Cases for Elastic Cloud on Kubernetes Bandwidth: Use Cases for Elastic Cloud on Kubernetes
Bandwidth: Use Cases for Elastic Cloud on Kubernetes
 

Mehr von Elasticsearch

An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxElasticsearch
 
From MSP to MSSP using Elastic
From MSP to MSSP using ElasticFrom MSP to MSSP using Elastic
From MSP to MSSP using ElasticElasticsearch
 
Cómo crear excelentes experiencias de búsqueda en sitios web
Cómo crear excelentes experiencias de búsqueda en sitios webCómo crear excelentes experiencias de búsqueda en sitios web
Cómo crear excelentes experiencias de búsqueda en sitios webElasticsearch
 
Te damos la bienvenida a una nueva forma de realizar búsquedas
Te damos la bienvenida a una nueva forma de realizar búsquedas Te damos la bienvenida a una nueva forma de realizar búsquedas
Te damos la bienvenida a una nueva forma de realizar búsquedas Elasticsearch
 
Tirez pleinement parti d'Elastic grâce à Elastic Cloud
Tirez pleinement parti d'Elastic grâce à Elastic CloudTirez pleinement parti d'Elastic grâce à Elastic Cloud
Tirez pleinement parti d'Elastic grâce à Elastic CloudElasticsearch
 
Comment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesComment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesElasticsearch
 
Plongez au cœur de la recherche dans tous ses états.
Plongez au cœur de la recherche dans tous ses états.Plongez au cœur de la recherche dans tous ses états.
Plongez au cœur de la recherche dans tous ses états.Elasticsearch
 
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]Elasticsearch
 
An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxElasticsearch
 
Welcome to a new state of find
Welcome to a new state of findWelcome to a new state of find
Welcome to a new state of findElasticsearch
 
Building great website search experiences
Building great website search experiencesBuilding great website search experiences
Building great website search experiencesElasticsearch
 
Keynote: Harnessing the power of Elasticsearch for simplified search
Keynote: Harnessing the power of Elasticsearch for simplified searchKeynote: Harnessing the power of Elasticsearch for simplified search
Keynote: Harnessing the power of Elasticsearch for simplified searchElasticsearch
 
Cómo transformar los datos en análisis con los que tomar decisiones
Cómo transformar los datos en análisis con los que tomar decisionesCómo transformar los datos en análisis con los que tomar decisiones
Cómo transformar los datos en análisis con los que tomar decisionesElasticsearch
 
Explore relève les défis Big Data avec Elastic Cloud
Explore relève les défis Big Data avec Elastic Cloud Explore relève les défis Big Data avec Elastic Cloud
Explore relève les défis Big Data avec Elastic Cloud Elasticsearch
 
Comment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesComment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesElasticsearch
 
Transforming data into actionable insights
Transforming data into actionable insightsTransforming data into actionable insights
Transforming data into actionable insightsElasticsearch
 
Opening Keynote: Why Elastic?
Opening Keynote: Why Elastic?Opening Keynote: Why Elastic?
Opening Keynote: Why Elastic?Elasticsearch
 
Empowering agencies using Elastic as a Service inside Government
Empowering agencies using Elastic as a Service inside GovernmentEmpowering agencies using Elastic as a Service inside Government
Empowering agencies using Elastic as a Service inside GovernmentElasticsearch
 
The opportunities and challenges of data for public good
The opportunities and challenges of data for public goodThe opportunities and challenges of data for public good
The opportunities and challenges of data for public goodElasticsearch
 
Enterprise search and unstructured data with CGI and Elastic
Enterprise search and unstructured data with CGI and ElasticEnterprise search and unstructured data with CGI and Elastic
Enterprise search and unstructured data with CGI and ElasticElasticsearch
 

Mehr von Elasticsearch (20)

An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolbox
 
From MSP to MSSP using Elastic
From MSP to MSSP using ElasticFrom MSP to MSSP using Elastic
From MSP to MSSP using Elastic
 
Cómo crear excelentes experiencias de búsqueda en sitios web
Cómo crear excelentes experiencias de búsqueda en sitios webCómo crear excelentes experiencias de búsqueda en sitios web
Cómo crear excelentes experiencias de búsqueda en sitios web
 
Te damos la bienvenida a una nueva forma de realizar búsquedas
Te damos la bienvenida a una nueva forma de realizar búsquedas Te damos la bienvenida a una nueva forma de realizar búsquedas
Te damos la bienvenida a una nueva forma de realizar búsquedas
 
Tirez pleinement parti d'Elastic grâce à Elastic Cloud
Tirez pleinement parti d'Elastic grâce à Elastic CloudTirez pleinement parti d'Elastic grâce à Elastic Cloud
Tirez pleinement parti d'Elastic grâce à Elastic Cloud
 
Comment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesComment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitables
 
Plongez au cœur de la recherche dans tous ses états.
Plongez au cœur de la recherche dans tous ses états.Plongez au cœur de la recherche dans tous ses états.
Plongez au cœur de la recherche dans tous ses états.
 
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
 
An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolbox
 
Welcome to a new state of find
Welcome to a new state of findWelcome to a new state of find
Welcome to a new state of find
 
Building great website search experiences
Building great website search experiencesBuilding great website search experiences
Building great website search experiences
 
Keynote: Harnessing the power of Elasticsearch for simplified search
Keynote: Harnessing the power of Elasticsearch for simplified searchKeynote: Harnessing the power of Elasticsearch for simplified search
Keynote: Harnessing the power of Elasticsearch for simplified search
 
Cómo transformar los datos en análisis con los que tomar decisiones
Cómo transformar los datos en análisis con los que tomar decisionesCómo transformar los datos en análisis con los que tomar decisiones
Cómo transformar los datos en análisis con los que tomar decisiones
 
Explore relève les défis Big Data avec Elastic Cloud
Explore relève les défis Big Data avec Elastic Cloud Explore relève les défis Big Data avec Elastic Cloud
Explore relève les défis Big Data avec Elastic Cloud
 
Comment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesComment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitables
 
Transforming data into actionable insights
Transforming data into actionable insightsTransforming data into actionable insights
Transforming data into actionable insights
 
Opening Keynote: Why Elastic?
Opening Keynote: Why Elastic?Opening Keynote: Why Elastic?
Opening Keynote: Why Elastic?
 
Empowering agencies using Elastic as a Service inside Government
Empowering agencies using Elastic as a Service inside GovernmentEmpowering agencies using Elastic as a Service inside Government
Empowering agencies using Elastic as a Service inside Government
 
The opportunities and challenges of data for public good
The opportunities and challenges of data for public goodThe opportunities and challenges of data for public good
The opportunities and challenges of data for public good
 
Enterprise search and unstructured data with CGI and Elastic
Enterprise search and unstructured data with CGI and ElasticEnterprise search and unstructured data with CGI and Elastic
Enterprise search and unstructured data with CGI and Elastic
 

Kürzlich hochgeladen

How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsJoaquim Jorge
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
Evaluating the top large language models.pdf
Evaluating the top large language models.pdfEvaluating the top large language models.pdf
Evaluating the top large language models.pdfChristopherTHyatt
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfTech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfhans926745
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024The Digital Insurer
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreternaman860154
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 

Kürzlich hochgeladen (20)

How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
Evaluating the top large language models.pdf
Evaluating the top large language models.pdfEvaluating the top large language models.pdf
Evaluating the top large language models.pdf
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfTech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdf
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 

Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elastic Cloud Enterprise at Otto

  • 1. Christian Herweg Thomas Klinger Lukas Kucharski DIVIDE & CONQUER Logging Architecture in Distributed Ecosystems with Elastic Cloud Enterprise
  • 2. Elastic{ON} Tour: Frankfurt 2018 Facts & Figures 07.11.18 2 • Location: 
 Hamburg (head office) • Employees: 
 4,700 (all locations) • Sales 2016/2017: 
 2.9 billion euros OTTO campus in Hamburg
  • 3. Elastic{ON} Tour: Frankfurt 2018 OTTO – Number 1 in Fashion & Lifestyle* 07.11.18 3 *B2C mail order, GfK figures 2014 6.800
 brands (in-house and many premium third-party brands) Over 2,8 million items online Huge product portfolio from fashion and lifestyle to household appliances and multimedia, DIY, kitchens, furniture and toys 120 
 specialist catalogues The only ‘big book’ company to make the jump into the digital world 6 specialist online shops
  • 4. Elastic{ON} Tour: Frankfurt 2018 Business Domains Mirror the System Architecture Page Assembly Tesla ShopOffice AfterSales Search P13N Order User Authentication Tracking u.v.m. Code ownership results in high code quality ! Continuous Delivery 
 permits more than 800 deployments per week. ! Verticals develop and test features fast and independently ! schematic presentation Business phases of the order process frame the distributed and parallel development. The technical mirror to this phase model allows highest possible flexibility in business concept development. ! 07.11.18 4
  • 5. Elastic{ON} Tour: Frankfurt 2018 07.11.18 5 Step by Step Fragmentation of otto.de Dedicated Monolith Vertical Architecture Micro Services & „Cloud Readiness“ I n d e xI n d e x I n d e x I n d e x I n d e x I n d e x Serverless & Cloud 2011 df /var/log 2013
 ~3TB data 2015
 ~17TB data 2018
 up to 42TB data &
 splunk >
  • 6. Elastic{ON} Tour: Frankfurt 2018 DISTRIBUTED LOGGING @ AWS a.k.a. „How to enable tenants to log data“ 07.11.18 6
  • 7. Elastic{ON} Tour: Frankfurt 2018 Requirements • Security • Encryption (at rest, in transit) • Authentication & Authorization • Isolation of resources („multi-tenancy“) • Accessing other vertical‘s logs => Rethink classic operations model, become a service provider 07.11.18 7
  • 8. Elastic{ON} Tour: Frankfurt 2018 Core Principles • Multi tenancy • Shared responsibility • Security by design • Automation (Goal: Provision the logging platform during lunch break) 07.11.18 8
  • 9. Elastic{ON} Tour: Frankfurt 2018 Elasticsearch Deployment at Scale • AWS Elasticsearch (Cost, missing encryption) • Elasticsearch on EC2 (Technical overhead) • Elastic Cloud Enterprise (fits requirements) 07.11.18 9
  • 10. Elastic{ON} Tour: Frankfurt 2018 Challenges • AWS Cross Account actions • (Near-)Realtime processing of logs • Processing multiple data formats (JSON, Syslog ...) • Queueing input data for failure scenarios • Autoscaling Logstash • Automation of Elasticsearch cluster management • Keeping up with new features in the Elastic stack 07.11.18 10
  • 11. Elastic{ON} Tour: Frankfurt 2018 ELASTIC CLOUD ENTERPRISE a.k.a. „How to provision tons of Elasticsearch and Kibana clusters“ 07.11.18 11
  • 12. Elastic{ON} Tour: Frankfurt 2018 How Elastic Cloud Enterprise Helped Us • No need to build custom provisioning service for Elasticsearch and Kibana clusters • Provides security features via Elastic features (authentication, authorization, integration with LDAP) • Supports multiple Elastic stack versions • Easy to set up a basic installation • Customizable (stack packs, underlying EC2 instance) • Extensive API • Updates do not require downtime • Multi-tenancy 07.11.18 12
  • 13. Elastic{ON} Tour: Frankfurt 2018 Architecture 07.11.18 13 Key facts: • 2 Loadbalancers • 3 ECE Proxies • 3 ECE Coordinators • 21 ECE Allocators (i3.4xlarge) • 20+ Clusters • Dedicated Monitoring Cluster
  • 14. Elastic{ON} Tour: Frankfurt 2018 Challenges • Automated deployment • Stateful components in a cloud environment (allocators) • Troubleshooting ECE • No fine-grained authentication and authorization (LDAP, OAuth2) except readonly/root users => Custom tooling required 07.11.18 14
  • 15. Elastic{ON} Tour: Frankfurt 2018 LOGINGEST a.k.a. „How to move logs from one account to another“ 07.11.18 15
  • 16. Elastic{ON} Tour: Frankfurt 2018 Challenges • Cross account log ingestion • Pipeline performance & autoscaling • Finding a suitable input • S3, CloudWatch Logs (Logstash state management) • Kinesis, Kafka (too expensive) • S3 & SNS & SQS (logstash-input-s3-sns-sqs) • Pipeline management (in Kibana) 07.11.18 16
  • 17. Elastic{ON} Tour: Frankfurt 2018 Architecture Logingest 07.11.18 17
  • 18. Elastic{ON} Tour: Frankfurt 2018 Autoscaling logstash 07.11.18 18 • Autoscale between 30 and 110 Logstash containers in total • Scaling based on demand and cluster size • Manual intervention possible • consume unexpected peaks • stopping ingest in failure scenarios
  • 19. Elastic{ON} Tour: Frankfurt 2018 HOUSEKEEPING a.k.a. „How to keep your clusters clean“ 07.11.18 19
  • 20. Elastic{ON} Tour: Frankfurt 2018 Curator as a Service • Verticals provide configuration in Git • Synchronized to S3 • Master-worker architecture based on AWS Lambda, CloudWatch and SQS 07.11.18 20
  • 21. Elastic{ON} Tour: Frankfurt 2018 Challenges • Verticals are responsible for curator configuration • Lack of knowledge (both Curator and Elasticsearch) • Monitoring Curator • Tracking origins of failed runs (misconfiguration, internal failure) • Transparency for verticals • AWS Lambda limits (max. 3 minutes runtime) 07.11.18 21
  • 22. Elastic{ON} Tour: Frankfurt 2018 LESSONS LEARNED a.k.a. „Learning through failure“ 07.11.18 22
  • 23. Elastic{ON} Tour: Frankfurt 2018 Learnings • Spread knowledge about Elasticsearch with teams • Create a sustainable knowledgebase • Automation is essential • Reduce operational overhead • Have time to develop and introduce new features • Know your I/O limits and requirements • Scaling Logstash is not a trivial task 07.11.18 23
  • 24. Elastic{ON} Tour: Frankfurt 2018 BACKLOG a.k.a. „Still some work to do“ 07.11.18 24
  • 25. Elastic{ON} Tour: Frankfurt 2018 Next steps • Develop custom Logstash pipeline management solution • Move Housekeeping workers from AWS Lambda to AWS Fargate • Evaluate Index Lifecyle Management via Elasticsearch • Upgrade to ECE 2.0 • Leverage potential of ECE & Elastic features • Tenants use and know about Machine Learning & APM • ECE 2.0 features • When available: cross-cluster search 07.11.18 25
  • 26. Elastic{ON} Tour: Frankfurt 2018 WE ARE HIRING! https://www.otto.de/jobs 07.11.18 26
  • 27. Elastic{ON} Tour: Frankfurt 2018 THANK YOU FOR YOUR ATTENTION 07.11.18 27
  • 28. Elastic{ON} Tour: Frankfurt 2018 BACKUP 07.11.18 28
  • 29. Elastic{ON} Tour: Frankfurt 2018 History – The Beginning... 07.11.18 29
  • 30. K2 - Cloud Readiness 08.05.2018 30 Autonomous and team specific utilization of technology. Decoupling of interfaces and system components reduces architecture complexity and interdependences especially with service components. Perimeter protection is replaced by an integrated security concept, reducing vulnerability and allowing customised implementations. Cloud migration allows new sourcing and scaling models