SlideShare ist ein Scribd-Unternehmen logo
1 von 46
Downloaden Sie, um offline zu lesen
Resilience
From Theory to Practice
by:
Efim Dimenstein - Chief Architect
Ori Cohen - Lead Resilience Engineer
Jan 2016
What is Liveperson
Liveperson transforms the
connection between brands and
consumers.
1.5 M Visits concurrent
3BN Visits/month
200BN API calls/month
2 PB data
Our Scale
99.97% Uptime
6 Data Centers
1000+ physical servers
6000+ VMs
Our Production
Fast release cycle
~250 people R&D
Constant Innovation
Multiple Technologies
Our Engineering
interruptions per month
on average 33 :)
The Past
The Past
The Present
LiveEngage Platform
Composable
~100 services
We keep splitting
Much easier to scale
LiveEngage Platform
Services are grouped into types
The platform is divided into layers
LiveEngage Platform
Everything That
Can Go Wrong
Will Go Wrong
Resilience from Theory to Practice
Resilience Pyramid
DC
HW
SERVICE
COMPONENT
CODE
DC Resilience - Global
DC Resilience
Primary
Secondary
Service
Node1
NodeN
Node2
Node3
...
Service X
Service
Node1
NodeN
Node2
Node3
...
Service X
HA Functionality
Service Grouping
Administration&
Configuration
Real Time
Near Real Time
Offline
Resilience from Theory to Practice
Components
Solve once - reuse
The Glue
Level of abstraction
Isolates common problems
Components - Guidelines
Retries
Fallback
Cache
Resilience from Theory to Practice
@ ground level
trust
company
trust
engineers
and still evaluate
knowledge is power
tooling
testing
deployment
metrics
logs
E2E
ALERTING
untested
==
unreliable
but… ?
costeffective
visibility
incident
injection
testing
process
opt-in
resilience @ scale
● multi layered solution
● requires monitoring and testing
● ingrained in the company culture
● keep things simple
● trust and empower your engineers
● break stuff
Thank
you!
Q&A

Weitere ähnliche Inhalte

Was ist angesagt?

Micro Services Architecture
Micro Services ArchitectureMicro Services Architecture
Micro Services ArchitectureRanjan Baisak
 
URP? Excuse You! The Three Kafka Metrics You Need to Know
URP? Excuse You! The Three Kafka Metrics You Need to KnowURP? Excuse You! The Three Kafka Metrics You Need to Know
URP? Excuse You! The Three Kafka Metrics You Need to KnowTodd Palino
 
Thorben Lindhauer: Live Coding: Zeebe - Camunda Day San Francisco
Thorben Lindhauer: Live Coding: Zeebe - Camunda Day San FranciscoThorben Lindhauer: Live Coding: Zeebe - Camunda Day San Francisco
Thorben Lindhauer: Live Coding: Zeebe - Camunda Day San Franciscocamunda services GmbH
 
How to create custom dashboards in Elastic Search / Kibana with Performance V...
How to create custom dashboards in Elastic Search / Kibana with Performance V...How to create custom dashboards in Elastic Search / Kibana with Performance V...
How to create custom dashboards in Elastic Search / Kibana with Performance V...PerformanceVision (previously SecurActive)
 
Jay Kreps, Confluent | Kafka Summit SF 2019 Keynote ft. Dev Tagare, Lyft + Pr...
Jay Kreps, Confluent | Kafka Summit SF 2019 Keynote ft. Dev Tagare, Lyft + Pr...Jay Kreps, Confluent | Kafka Summit SF 2019 Keynote ft. Dev Tagare, Lyft + Pr...
Jay Kreps, Confluent | Kafka Summit SF 2019 Keynote ft. Dev Tagare, Lyft + Pr...confluent
 
"Why we all build bad architectures and how to stop doing it", Vova Kyrychenko
"Why we all build bad architectures and how to stop doing it", Vova Kyrychenko"Why we all build bad architectures and how to stop doing it", Vova Kyrychenko
"Why we all build bad architectures and how to stop doing it", Vova KyrychenkoFwdays
 
Monitoring Serverless Applications with Datadog
Monitoring Serverless Applications with DatadogMonitoring Serverless Applications with Datadog
Monitoring Serverless Applications with DatadogDevOps.com
 
Zeebe 0.20.0 Release Webinar - July 22, 2019
Zeebe 0.20.0 Release Webinar - July 22, 2019Zeebe 0.20.0 Release Webinar - July 22, 2019
Zeebe 0.20.0 Release Webinar - July 22, 2019Zeebe
 
PayPal Risk Platform High Performance Practice
PayPal Risk Platform High Performance PracticePayPal Risk Platform High Performance Practice
PayPal Risk Platform High Performance PracticeBrian Ling
 
The eBay Architecture: Striking a Balance between Site Stability, Feature Ve...
The eBay Architecture:  Striking a Balance between Site Stability, Feature Ve...The eBay Architecture:  Striking a Balance between Site Stability, Feature Ve...
The eBay Architecture: Striking a Balance between Site Stability, Feature Ve...Randy Shoup
 
"Architect in Consulting: all you wanted to know about "Discovery"", Dmytro O...
"Architect in Consulting: all you wanted to know about "Discovery"", Dmytro O..."Architect in Consulting: all you wanted to know about "Discovery"", Dmytro O...
"Architect in Consulting: all you wanted to know about "Discovery"", Dmytro O...Fwdays
 
From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous M...
From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous M...From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous M...
From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous M...confluent
 
Microsoft challenges of a multi tenant kafka service
Microsoft challenges of a multi tenant kafka serviceMicrosoft challenges of a multi tenant kafka service
Microsoft challenges of a multi tenant kafka serviceNitin Kumar
 
Redis Streams for Event-Driven Microservices
Redis Streams for Event-Driven MicroservicesRedis Streams for Event-Driven Microservices
Redis Streams for Event-Driven MicroservicesRedis Labs
 
How we eased out security journey with OAuth (Goodbye Kerberos!) | Paul Makka...
How we eased out security journey with OAuth (Goodbye Kerberos!) | Paul Makka...How we eased out security journey with OAuth (Goodbye Kerberos!) | Paul Makka...
How we eased out security journey with OAuth (Goodbye Kerberos!) | Paul Makka...HostedbyConfluent
 
Observability with Spring-based distributed systems
Observability with Spring-based distributed systemsObservability with Spring-based distributed systems
Observability with Spring-based distributed systemsRakuten Group, Inc.
 
How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...
How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...
How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...Lightbend
 
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesPatterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesJosef Adersberger
 

Was ist angesagt? (20)

Micro Services Architecture
Micro Services ArchitectureMicro Services Architecture
Micro Services Architecture
 
URP? Excuse You! The Three Kafka Metrics You Need to Know
URP? Excuse You! The Three Kafka Metrics You Need to KnowURP? Excuse You! The Three Kafka Metrics You Need to Know
URP? Excuse You! The Three Kafka Metrics You Need to Know
 
Thorben Lindhauer: Live Coding: Zeebe - Camunda Day San Francisco
Thorben Lindhauer: Live Coding: Zeebe - Camunda Day San FranciscoThorben Lindhauer: Live Coding: Zeebe - Camunda Day San Francisco
Thorben Lindhauer: Live Coding: Zeebe - Camunda Day San Francisco
 
How to create custom dashboards in Elastic Search / Kibana with Performance V...
How to create custom dashboards in Elastic Search / Kibana with Performance V...How to create custom dashboards in Elastic Search / Kibana with Performance V...
How to create custom dashboards in Elastic Search / Kibana with Performance V...
 
Jay Kreps, Confluent | Kafka Summit SF 2019 Keynote ft. Dev Tagare, Lyft + Pr...
Jay Kreps, Confluent | Kafka Summit SF 2019 Keynote ft. Dev Tagare, Lyft + Pr...Jay Kreps, Confluent | Kafka Summit SF 2019 Keynote ft. Dev Tagare, Lyft + Pr...
Jay Kreps, Confluent | Kafka Summit SF 2019 Keynote ft. Dev Tagare, Lyft + Pr...
 
"Why we all build bad architectures and how to stop doing it", Vova Kyrychenko
"Why we all build bad architectures and how to stop doing it", Vova Kyrychenko"Why we all build bad architectures and how to stop doing it", Vova Kyrychenko
"Why we all build bad architectures and how to stop doing it", Vova Kyrychenko
 
Monitoring Serverless Applications with Datadog
Monitoring Serverless Applications with DatadogMonitoring Serverless Applications with Datadog
Monitoring Serverless Applications with Datadog
 
Zeebe 0.20.0 Release Webinar - July 22, 2019
Zeebe 0.20.0 Release Webinar - July 22, 2019Zeebe 0.20.0 Release Webinar - July 22, 2019
Zeebe 0.20.0 Release Webinar - July 22, 2019
 
PayPal Risk Platform High Performance Practice
PayPal Risk Platform High Performance PracticePayPal Risk Platform High Performance Practice
PayPal Risk Platform High Performance Practice
 
The eBay Architecture: Striking a Balance between Site Stability, Feature Ve...
The eBay Architecture:  Striking a Balance between Site Stability, Feature Ve...The eBay Architecture:  Striking a Balance between Site Stability, Feature Ve...
The eBay Architecture: Striking a Balance between Site Stability, Feature Ve...
 
"Architect in Consulting: all you wanted to know about "Discovery"", Dmytro O...
"Architect in Consulting: all you wanted to know about "Discovery"", Dmytro O..."Architect in Consulting: all you wanted to know about "Discovery"", Dmytro O...
"Architect in Consulting: all you wanted to know about "Discovery"", Dmytro O...
 
Workshop on CIFS / SMB Protocol Performance Analysis
Workshop on CIFS / SMB Protocol Performance AnalysisWorkshop on CIFS / SMB Protocol Performance Analysis
Workshop on CIFS / SMB Protocol Performance Analysis
 
From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous M...
From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous M...From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous M...
From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous M...
 
Twelve Factor App
Twelve Factor AppTwelve Factor App
Twelve Factor App
 
Microsoft challenges of a multi tenant kafka service
Microsoft challenges of a multi tenant kafka serviceMicrosoft challenges of a multi tenant kafka service
Microsoft challenges of a multi tenant kafka service
 
Redis Streams for Event-Driven Microservices
Redis Streams for Event-Driven MicroservicesRedis Streams for Event-Driven Microservices
Redis Streams for Event-Driven Microservices
 
How we eased out security journey with OAuth (Goodbye Kerberos!) | Paul Makka...
How we eased out security journey with OAuth (Goodbye Kerberos!) | Paul Makka...How we eased out security journey with OAuth (Goodbye Kerberos!) | Paul Makka...
How we eased out security journey with OAuth (Goodbye Kerberos!) | Paul Makka...
 
Observability with Spring-based distributed systems
Observability with Spring-based distributed systemsObservability with Spring-based distributed systems
Observability with Spring-based distributed systems
 
How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...
How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...
How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...
 
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesPatterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to Kubernetes
 

Andere mochten auch

Measure() or die()
Measure() or die() Measure() or die()
Measure() or die() LivePerson
 
Functional programming with Java 8
Functional programming with Java 8Functional programming with Java 8
Functional programming with Java 8LivePerson
 
Kubernetes your tests! automation with docker on google cloud platform
Kubernetes your tests! automation with docker on google cloud platformKubernetes your tests! automation with docker on google cloud platform
Kubernetes your tests! automation with docker on google cloud platformLivePerson
 
Promoting Resilience for Children and Young People in Relation to Social Medi...
Promoting Resilience for Children and Young People in Relation to Social Medi...Promoting Resilience for Children and Young People in Relation to Social Medi...
Promoting Resilience for Children and Young People in Relation to Social Medi...Bex Lewis
 
resilience personal_presentation V _headquarters - NO NOTES
resilience personal_presentation V _headquarters - NO NOTESresilience personal_presentation V _headquarters - NO NOTES
resilience personal_presentation V _headquarters - NO NOTESAmanda Comoretto
 
08 stress, coping, trauma, resilience
08   stress, coping, trauma, resilience08   stress, coping, trauma, resilience
08 stress, coping, trauma, resilienceDickson College
 
Dr. Kenneth Ginsburg - Building Resilience
Dr. Kenneth Ginsburg - Building ResilienceDr. Kenneth Ginsburg - Building Resilience
Dr. Kenneth Ginsburg - Building ResilienceCommunity Parent Network
 
Developing Your Resilience
Developing Your ResilienceDeveloping Your Resilience
Developing Your ResilienceJeffrey Russell
 
Graph QL Introduction
Graph QL IntroductionGraph QL Introduction
Graph QL IntroductionLivePerson
 
Without Resilience, Nothing Else Matters
Without Resilience, Nothing Else MattersWithout Resilience, Nothing Else Matters
Without Resilience, Nothing Else MattersJonas Bonér
 
Resilience: how to build resilience in your people and your organization
Resilience: how to build resilience in your people and your organizationResilience: how to build resilience in your people and your organization
Resilience: how to build resilience in your people and your organizationDelta Partners
 
Develop Your Personal Resilience!
Develop Your Personal Resilience!Develop Your Personal Resilience!
Develop Your Personal Resilience!Charlotte Waller
 
Resilience presentation
Resilience presentationResilience presentation
Resilience presentationJoe Krause
 
Weblogic as a windows service
Weblogic as a windows serviceWeblogic as a windows service
Weblogic as a windows serviceRavi Kumar Lanke
 

Andere mochten auch (20)

Measure() or die()
Measure() or die() Measure() or die()
Measure() or die()
 
Functional programming with Java 8
Functional programming with Java 8Functional programming with Java 8
Functional programming with Java 8
 
Kubernetes your tests! automation with docker on google cloud platform
Kubernetes your tests! automation with docker on google cloud platformKubernetes your tests! automation with docker on google cloud platform
Kubernetes your tests! automation with docker on google cloud platform
 
Promoting Resilience for Children and Young People in Relation to Social Medi...
Promoting Resilience for Children and Young People in Relation to Social Medi...Promoting Resilience for Children and Young People in Relation to Social Medi...
Promoting Resilience for Children and Young People in Relation to Social Medi...
 
resilience personal_presentation V _headquarters - NO NOTES
resilience personal_presentation V _headquarters - NO NOTESresilience personal_presentation V _headquarters - NO NOTES
resilience personal_presentation V _headquarters - NO NOTES
 
Resilience
ResilienceResilience
Resilience
 
08 stress, coping, trauma, resilience
08   stress, coping, trauma, resilience08   stress, coping, trauma, resilience
08 stress, coping, trauma, resilience
 
Dr. Kenneth Ginsburg - Building Resilience
Dr. Kenneth Ginsburg - Building ResilienceDr. Kenneth Ginsburg - Building Resilience
Dr. Kenneth Ginsburg - Building Resilience
 
Developing Your Resilience
Developing Your ResilienceDeveloping Your Resilience
Developing Your Resilience
 
Resilience and empowerment
Resilience and empowermentResilience and empowerment
Resilience and empowerment
 
Resilience ppt
Resilience pptResilience ppt
Resilience ppt
 
Graph QL Introduction
Graph QL IntroductionGraph QL Introduction
Graph QL Introduction
 
Without Resilience, Nothing Else Matters
Without Resilience, Nothing Else MattersWithout Resilience, Nothing Else Matters
Without Resilience, Nothing Else Matters
 
Resilience: how to build resilience in your people and your organization
Resilience: how to build resilience in your people and your organizationResilience: how to build resilience in your people and your organization
Resilience: how to build resilience in your people and your organization
 
Build your resilience
Build your resilience Build your resilience
Build your resilience
 
Develop Your Personal Resilience!
Develop Your Personal Resilience!Develop Your Personal Resilience!
Develop Your Personal Resilience!
 
Resilience presentation
Resilience presentationResilience presentation
Resilience presentation
 
Patterns of resilience
Patterns of resiliencePatterns of resilience
Patterns of resilience
 
Weblogic as a windows service
Weblogic as a windows serviceWeblogic as a windows service
Weblogic as a windows service
 
Noticias TIC octubre 2012
Noticias TIC octubre  2012Noticias TIC octubre  2012
Noticias TIC octubre 2012
 

Ähnlich wie Resilience from Theory to Practice

Office 365 introduction
Office 365 introductionOffice 365 introduction
Office 365 introductionGlobal News
 
Lunch and Learn and Sneakers
Lunch and Learn and SneakersLunch and Learn and Sneakers
Lunch and Learn and SneakersBill Zajac
 
Cisco Connect 2018 Thailand - Enabling the next gen data center transformatio...
Cisco Connect 2018 Thailand - Enabling the next gen data center transformatio...Cisco Connect 2018 Thailand - Enabling the next gen data center transformatio...
Cisco Connect 2018 Thailand - Enabling the next gen data center transformatio...NetworkCollaborators
 
Netflix Edge Engineering Open House Presentations - June 9, 2016
Netflix Edge Engineering Open House Presentations - June 9, 2016Netflix Edge Engineering Open House Presentations - June 9, 2016
Netflix Edge Engineering Open House Presentations - June 9, 2016Daniel Jacobson
 
Meetup Microservices Commandments
Meetup Microservices CommandmentsMeetup Microservices Commandments
Meetup Microservices CommandmentsBill Zajac
 
Scaling Slack - The Good, the Unexpected, and the Road Ahead
Scaling Slack - The Good, the Unexpected, and the Road AheadScaling Slack - The Good, the Unexpected, and the Road Ahead
Scaling Slack - The Good, the Unexpected, and the Road AheadC4Media
 
5 Years Of Building SaaS On AWS
5 Years Of Building SaaS On AWS5 Years Of Building SaaS On AWS
5 Years Of Building SaaS On AWSChristian Beedgen
 
XenApp Virtualization Success - How to Eliminate Storage and Networking Bottl...
XenApp Virtualization Success - How to Eliminate Storage and Networking Bottl...XenApp Virtualization Success - How to Eliminate Storage and Networking Bottl...
XenApp Virtualization Success - How to Eliminate Storage and Networking Bottl...eG Innovations
 
Partner Event with ChunTai Tech Industry in Taipei - Oct 2017
Partner Event with ChunTai Tech Industry in Taipei - Oct 2017Partner Event with ChunTai Tech Industry in Taipei - Oct 2017
Partner Event with ChunTai Tech Industry in Taipei - Oct 2017Christoph Adler
 
QConSF2016-JoshEvans-MasteringChaosANetflixGuidetoMicroservices-compressed.pdf
QConSF2016-JoshEvans-MasteringChaosANetflixGuidetoMicroservices-compressed.pdfQConSF2016-JoshEvans-MasteringChaosANetflixGuidetoMicroservices-compressed.pdf
QConSF2016-JoshEvans-MasteringChaosANetflixGuidetoMicroservices-compressed.pdfSimranjyotSuri
 
Architectural Commandments for Building & Running Microservices at Scale
Architectural Commandments for Building & Running Microservices at ScaleArchitectural Commandments for Building & Running Microservices at Scale
Architectural Commandments for Building & Running Microservices at ScaleBrian Wilson
 
Mastering Chaos - A Netflix Guide to Microservices
Mastering Chaos - A Netflix Guide to MicroservicesMastering Chaos - A Netflix Guide to Microservices
Mastering Chaos - A Netflix Guide to MicroservicesJosh Evans
 
Cisco on Cisco. Defining the NG Cloud & Data Center Services
Cisco on Cisco. Defining the NG Cloud & Data Center ServicesCisco on Cisco. Defining the NG Cloud & Data Center Services
Cisco on Cisco. Defining the NG Cloud & Data Center ServicesCisco Russia
 
VmWare - 14oct2010
VmWare - 14oct2010VmWare - 14oct2010
VmWare - 14oct2010Agora Group
 
Redis Enterprise for Large-Scale Deployment - ENT224 - re:Invent 2017
Redis Enterprise for Large-Scale Deployment - ENT224 - re:Invent 2017Redis Enterprise for Large-Scale Deployment - ENT224 - re:Invent 2017
Redis Enterprise for Large-Scale Deployment - ENT224 - re:Invent 2017Amazon Web Services
 
The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato ...
The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato ...The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato ...
The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato ...John Viner
 
Juniper Corporate Presentation
Juniper Corporate PresentationJuniper Corporate Presentation
Juniper Corporate Presentationmauthay
 
VMworld 2013: US Air National Guard - DoD Private Cloud Initiative –How Virtu...
VMworld 2013: US Air National Guard - DoD Private Cloud Initiative –How Virtu...VMworld 2013: US Air National Guard - DoD Private Cloud Initiative –How Virtu...
VMworld 2013: US Air National Guard - DoD Private Cloud Initiative –How Virtu...VMworld
 
DevOps Transformation at Dynatrace and with Dynatrace
DevOps Transformation at Dynatrace and with DynatraceDevOps Transformation at Dynatrace and with Dynatrace
DevOps Transformation at Dynatrace and with DynatraceAndreas Grabner
 

Ähnlich wie Resilience from Theory to Practice (20)

Office 365 introduction
Office 365 introductionOffice 365 introduction
Office 365 introduction
 
Lunch and Learn and Sneakers
Lunch and Learn and SneakersLunch and Learn and Sneakers
Lunch and Learn and Sneakers
 
Cisco Connect 2018 Thailand - Enabling the next gen data center transformatio...
Cisco Connect 2018 Thailand - Enabling the next gen data center transformatio...Cisco Connect 2018 Thailand - Enabling the next gen data center transformatio...
Cisco Connect 2018 Thailand - Enabling the next gen data center transformatio...
 
Netflix Edge Engineering Open House Presentations - June 9, 2016
Netflix Edge Engineering Open House Presentations - June 9, 2016Netflix Edge Engineering Open House Presentations - June 9, 2016
Netflix Edge Engineering Open House Presentations - June 9, 2016
 
Meetup Microservices Commandments
Meetup Microservices CommandmentsMeetup Microservices Commandments
Meetup Microservices Commandments
 
Scaling Slack - The Good, the Unexpected, and the Road Ahead
Scaling Slack - The Good, the Unexpected, and the Road AheadScaling Slack - The Good, the Unexpected, and the Road Ahead
Scaling Slack - The Good, the Unexpected, and the Road Ahead
 
5 Years Of Building SaaS On AWS
5 Years Of Building SaaS On AWS5 Years Of Building SaaS On AWS
5 Years Of Building SaaS On AWS
 
XenApp Virtualization Success - How to Eliminate Storage and Networking Bottl...
XenApp Virtualization Success - How to Eliminate Storage and Networking Bottl...XenApp Virtualization Success - How to Eliminate Storage and Networking Bottl...
XenApp Virtualization Success - How to Eliminate Storage and Networking Bottl...
 
Partner Event with ChunTai Tech Industry in Taipei - Oct 2017
Partner Event with ChunTai Tech Industry in Taipei - Oct 2017Partner Event with ChunTai Tech Industry in Taipei - Oct 2017
Partner Event with ChunTai Tech Industry in Taipei - Oct 2017
 
QConSF2016-JoshEvans-MasteringChaosANetflixGuidetoMicroservices-compressed.pdf
QConSF2016-JoshEvans-MasteringChaosANetflixGuidetoMicroservices-compressed.pdfQConSF2016-JoshEvans-MasteringChaosANetflixGuidetoMicroservices-compressed.pdf
QConSF2016-JoshEvans-MasteringChaosANetflixGuidetoMicroservices-compressed.pdf
 
Architectural Commandments for Building & Running Microservices at Scale
Architectural Commandments for Building & Running Microservices at ScaleArchitectural Commandments for Building & Running Microservices at Scale
Architectural Commandments for Building & Running Microservices at Scale
 
Mastering Chaos - A Netflix Guide to Microservices
Mastering Chaos - A Netflix Guide to MicroservicesMastering Chaos - A Netflix Guide to Microservices
Mastering Chaos - A Netflix Guide to Microservices
 
Cisco on Cisco. Defining the NG Cloud & Data Center Services
Cisco on Cisco. Defining the NG Cloud & Data Center ServicesCisco on Cisco. Defining the NG Cloud & Data Center Services
Cisco on Cisco. Defining the NG Cloud & Data Center Services
 
VmWare - 14oct2010
VmWare - 14oct2010VmWare - 14oct2010
VmWare - 14oct2010
 
Redis Enterprise for Large-Scale Deployment - ENT224 - re:Invent 2017
Redis Enterprise for Large-Scale Deployment - ENT224 - re:Invent 2017Redis Enterprise for Large-Scale Deployment - ENT224 - re:Invent 2017
Redis Enterprise for Large-Scale Deployment - ENT224 - re:Invent 2017
 
The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato ...
The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato ...The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato ...
The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato ...
 
Juniper Corporate Presentation
Juniper Corporate PresentationJuniper Corporate Presentation
Juniper Corporate Presentation
 
Shift left Observability
Shift left ObservabilityShift left Observability
Shift left Observability
 
VMworld 2013: US Air National Guard - DoD Private Cloud Initiative –How Virtu...
VMworld 2013: US Air National Guard - DoD Private Cloud Initiative –How Virtu...VMworld 2013: US Air National Guard - DoD Private Cloud Initiative –How Virtu...
VMworld 2013: US Air National Guard - DoD Private Cloud Initiative –How Virtu...
 
DevOps Transformation at Dynatrace and with Dynatrace
DevOps Transformation at Dynatrace and with DynatraceDevOps Transformation at Dynatrace and with Dynatrace
DevOps Transformation at Dynatrace and with Dynatrace
 

Mehr von LivePerson

Microservices on top of kafka
Microservices on top of kafkaMicroservices on top of kafka
Microservices on top of kafkaLivePerson
 
System Revolution- How We Did It
System Revolution- How We Did It System Revolution- How We Did It
System Revolution- How We Did It LivePerson
 
Http 2: Should I care?
Http 2: Should I care?Http 2: Should I care?
Http 2: Should I care?LivePerson
 
Mobile app real-time content modifications using websockets
Mobile app real-time content modifications using websocketsMobile app real-time content modifications using websockets
Mobile app real-time content modifications using websocketsLivePerson
 
Mobile SDK: Considerations & Best Practices
Mobile SDK: Considerations & Best Practices Mobile SDK: Considerations & Best Practices
Mobile SDK: Considerations & Best Practices LivePerson
 
Apache Avro in LivePerson [Hebrew]
Apache Avro in LivePerson [Hebrew]Apache Avro in LivePerson [Hebrew]
Apache Avro in LivePerson [Hebrew]LivePerson
 
Apache Avro and Messaging at Scale in LivePerson
Apache Avro and Messaging at Scale in LivePersonApache Avro and Messaging at Scale in LivePerson
Apache Avro and Messaging at Scale in LivePersonLivePerson
 
Data compression in Modern Application
Data compression in Modern ApplicationData compression in Modern Application
Data compression in Modern ApplicationLivePerson
 
Support Office Hour Webinar - LivePerson API
Support Office Hour Webinar - LivePerson API Support Office Hour Webinar - LivePerson API
Support Office Hour Webinar - LivePerson API LivePerson
 
SIP - Introduction to SIP Protocol
SIP - Introduction to SIP ProtocolSIP - Introduction to SIP Protocol
SIP - Introduction to SIP ProtocolLivePerson
 
Scalding: Reaching Efficient MapReduce
Scalding: Reaching Efficient MapReduceScalding: Reaching Efficient MapReduce
Scalding: Reaching Efficient MapReduceLivePerson
 
Building Enterprise Level End-To-End Monitor System with Open Source Solution...
Building Enterprise Level End-To-End Monitor System with Open Source Solution...Building Enterprise Level End-To-End Monitor System with Open Source Solution...
Building Enterprise Level End-To-End Monitor System with Open Source Solution...LivePerson
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data ScienceLivePerson
 
From a Kafkaesque Story to The Promised Land at LivePerson
From a Kafkaesque Story to The Promised Land at LivePersonFrom a Kafkaesque Story to The Promised Land at LivePerson
From a Kafkaesque Story to The Promised Land at LivePersonLivePerson
 
How can A/B testing go wrong?
How can A/B testing go wrong?How can A/B testing go wrong?
How can A/B testing go wrong?LivePerson
 
Introduction to Vertica (Architecture & More)
Introduction to Vertica (Architecture & More)Introduction to Vertica (Architecture & More)
Introduction to Vertica (Architecture & More)LivePerson
 

Mehr von LivePerson (16)

Microservices on top of kafka
Microservices on top of kafkaMicroservices on top of kafka
Microservices on top of kafka
 
System Revolution- How We Did It
System Revolution- How We Did It System Revolution- How We Did It
System Revolution- How We Did It
 
Http 2: Should I care?
Http 2: Should I care?Http 2: Should I care?
Http 2: Should I care?
 
Mobile app real-time content modifications using websockets
Mobile app real-time content modifications using websocketsMobile app real-time content modifications using websockets
Mobile app real-time content modifications using websockets
 
Mobile SDK: Considerations & Best Practices
Mobile SDK: Considerations & Best Practices Mobile SDK: Considerations & Best Practices
Mobile SDK: Considerations & Best Practices
 
Apache Avro in LivePerson [Hebrew]
Apache Avro in LivePerson [Hebrew]Apache Avro in LivePerson [Hebrew]
Apache Avro in LivePerson [Hebrew]
 
Apache Avro and Messaging at Scale in LivePerson
Apache Avro and Messaging at Scale in LivePersonApache Avro and Messaging at Scale in LivePerson
Apache Avro and Messaging at Scale in LivePerson
 
Data compression in Modern Application
Data compression in Modern ApplicationData compression in Modern Application
Data compression in Modern Application
 
Support Office Hour Webinar - LivePerson API
Support Office Hour Webinar - LivePerson API Support Office Hour Webinar - LivePerson API
Support Office Hour Webinar - LivePerson API
 
SIP - Introduction to SIP Protocol
SIP - Introduction to SIP ProtocolSIP - Introduction to SIP Protocol
SIP - Introduction to SIP Protocol
 
Scalding: Reaching Efficient MapReduce
Scalding: Reaching Efficient MapReduceScalding: Reaching Efficient MapReduce
Scalding: Reaching Efficient MapReduce
 
Building Enterprise Level End-To-End Monitor System with Open Source Solution...
Building Enterprise Level End-To-End Monitor System with Open Source Solution...Building Enterprise Level End-To-End Monitor System with Open Source Solution...
Building Enterprise Level End-To-End Monitor System with Open Source Solution...
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
From a Kafkaesque Story to The Promised Land at LivePerson
From a Kafkaesque Story to The Promised Land at LivePersonFrom a Kafkaesque Story to The Promised Land at LivePerson
From a Kafkaesque Story to The Promised Land at LivePerson
 
How can A/B testing go wrong?
How can A/B testing go wrong?How can A/B testing go wrong?
How can A/B testing go wrong?
 
Introduction to Vertica (Architecture & More)
Introduction to Vertica (Architecture & More)Introduction to Vertica (Architecture & More)
Introduction to Vertica (Architecture & More)
 

Kürzlich hochgeladen

Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 

Kürzlich hochgeladen (20)

Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 

Resilience from Theory to Practice

Hinweis der Redaktion

  1. SaaS
  2. ?
  3. brief of legacy system Monolithic system Active Passive Can be made resilient but ….
  4. LiveEngage Micro Services Oriented Composable Simple Reusable Systems Scalable Flexible Decoupled
  5. Go into more detail
  6. Looked from the top Built a Resiliency pyramid
  7. Global Services Easiest - AA
  8. Sticky services - AA or 2 way AP
  9. Reduce a lot of services into several types Group them into layers - Biz. critical -> Mission Critical Focus on mission critical Go in order of priority Dependency only downward
  10. Service dependencies Unpredictable behaviours Domino failures Partial failures
  11. Retries Define a retry mechanism (Client to server & server to server) Never give up - recovery after failure Beware DDOS on yourself Fallback Fake it until you make it provide a fallback Scoring - flip a coin, use previous value, tell the client to come back soon write somewhere else (file instead of DB) Cache simplest resilience technique might be used as a fallback or as an abstraction level over a service
  12. up to now in the talk we talked about things from 30K feet up time to descend to ground level let’s see what is required on a day to day basis to make this work so resilience requires an ongoing never-ending effort
  13. get company wide buy-in show the current damage without pointing fingers at anyone
  14. trust your engineers u can’t do it alone! example: lessons-learned sessions including… + follow up
  15. architectural resilience evaluation of design periodic re-evaluation of services with score-cards
  16. provide R&D training during new employee onboarding encourage transition from Java to functional immutable code using Scala
  17. bi-monthly meetings with tech-leads and architects resilience guidelines
  18. CI and test environments to simulate production E2E tests run 24/7 to make sure entire system works every build passes through CI every release is first passed through a canary like prod. env. before GA
  19. dev teams manage their services’ deployment packages operations deploy to clusters changes made only through deployments no work and changes on a per machine basis
  20. remember the scale Efim mentioned in the start of the talk? metrics collected in realtime & processed by Zabbix
  21. pushed in realtime from web and app via Kafka to logstash -> ElasticSearch
  22. ±250 tests run over Jenkins user experience monitoring errors includes video of UI historical data is saved to ElasticSearch and presented...
  23. Support Tier1-3, NOC, ScS, experts, monitoring, E2E visual dashboards showing data of processed information bit from all inputs
  24. very early on we realized: best preparation for worst => break things you don’t want to rely on statements of your engineers “everything will be ok”
  25. where to test? what to test for? when and how often to test? system and service readiness
  26. concentrate on stuff that happens most in production the big important things (they matter the most) the low hanging fruit (easy wins) where to start? => mission critical, business critical base on data-flow initial focus on clients
  27. so we want to break things… OMG how do we get visibility inside and high resolution and granularity API monitors: as small as can be run at high frequency broad coverage of perimeter services (user experience)
  28. vmware based automation vmware API-M E2E app & web logs Kafka -> logstash -> ElasticSearch -> Kibana runs on test env. the entire system runs there
  29. process => nights + weekends + holidays results complicated scenarios that can’t reproduce in env are tested in production DR
  30. teams need to opt-in process
  31. if you are looking to get resilience at a large scale, keep those things in mind multi layered solution requires monitoring and testing you need resilience to be ingrained in the culture of your company in all levels to get not support and cooperation but contribution and initiative remember to keep things simple trust and empower your engineers and go break stuff