Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

When Developers Operate and Operators Develop

9.403 Aufrufe

Veröffentlicht am

Businesses are speeding up development and automating operations to remain competitive and to get large organizations to scale. Project based monolithic application updates are replaced by product teams owning containerized microservices. This puts developers on call, responsible for pushing code to production, fixing it when it breaks, and managing the cost and security aspects of running their microservices. In this world operations skill-sets are either embedded in the microservices development teams, or  building and operating API driven platforms. The platform automates stress testing, canary based deployment, penetration testing and enforces availability and security requirements.  There are no meetings or tickets to file in the delivery process for updating a containerized microservice, which can happen many times a day, and takes seconds to complete. The role of site reliability engineering moves from firefighting and fixing outages to buiding tools for finding problems and routing those problems to the right developers. SREs manage the incident lifecycle for customer visible problems, and measure and publish availability metrics. This may sound futuristic but Werner Vogels described this as “You build it, you run it” in 2006.

Veröffentlicht in: Technologie
  • Do This Simple 2-Minute Ritual To Loss 1 Pound Of Belly Fat Every 72 Hours ●●● https://tinyurl.com/y6qaaou7
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

When Developers Operate and Operators Develop

  1. 1. When developers operate, and operators develop… Adrian Cockcroft @adrianco Technology Fellow - Battery Ventures June 2015
  2. 2. “You build it, you run it.” Werner Vogels 2006
  3. 3. Key Goals of the CIO? Align IT with the business Develop products faster Try not to get breached
  4. 4. Security Blanket Failure Insecure applications hidden behind firewalls make you feel safe until the breach happens… http://peanuts.wikia.com/wiki/Linus'_security_blanket
  5. 5. What needs to change?
  6. 6. Developer responsibilities: Faster, cheaper, safer
  7. 7. Operator responsibilities: API driven platform automation
  8. 8. Product Development Processes
  9. 9. Assumption: Process prevents problems
  10. 10. Organizations build up slow complex “Scar tissue” processes
  11. 11. Observe Orient Decide Act Continuous Delivery
  12. 12. Observe Orient Decide Act Land grab opportunity Competitive Move Customer Pain Point Measure Customers Continuous Delivery
  13. 13. Observe Orient Decide Act Land grab opportunity Competitive Move Customer Pain Point INNOVATION Measure Customers Continuous Delivery
  14. 14. Observe Orient Decide Act Land grab opportunity Competitive Move Customer Pain Point Analysis Model Hypotheses INNOVATION Measure Customers Continuous Delivery
  15. 15. Observe Orient Decide Act Land grab opportunity Competitive Move Customer Pain Point Analysis Model Hypotheses BIG DATA INNOVATION Measure Customers Continuous Delivery
  16. 16. Observe Orient Decide Act Land grab opportunity Competitive Move Customer Pain Point Analysis JFDI Plan Response Share Plans Model Hypotheses BIG DATA INNOVATION Measure Customers Continuous Delivery
  17. 17. Observe Orient Decide Act Land grab opportunity Competitive Move Customer Pain Point Analysis JFDI Plan Response Share Plans Model Hypotheses BIG DATA INNOVATION CULTURE Measure Customers Continuous Delivery
  18. 18. Observe Orient Decide Act Land grab opportunity Competitive Move Customer Pain Point Analysis JFDI Plan Response Share Plans Incremental Features Automatic Deploy Launch AB Test Model Hypotheses BIG DATA INNOVATION CULTURE Measure Customers Continuous Delivery
  19. 19. Observe Orient Decide Act Land grab opportunity Competitive Move Customer Pain Point Analysis JFDI Plan Response Share Plans Incremental Features Automatic Deploy Launch AB Test Model Hypotheses BIG DATA INNOVATION CULTURE CLOUD Measure Customers Continuous Delivery
  20. 20. Observe Orient Decide Act Land grab opportunity Competitive Move Customer Pain Point Analysis JFDI Plan Response Share Plans Incremental Features Automatic Deploy Launch AB Test Model Hypotheses BIG DATA INNOVATION CULTURE CLOUD Measure Customers Continuous Delivery
  21. 21. Observe Orient Decide Act Land grab opportunity Competitive Move Customer Pain Point Analysis JFDI Plan Response Share Plans Incremental Features Automatic Deploy Launch AB Test Model Hypotheses BIG DATA INNOVATION CULTURE CLOUD Measure Customers Continuous Delivery
  22. 22. Breaking Down the SILOs
  23. 23. Breaking Down the SILOs QA DBA Sys Adm Net Adm SAN Adm DevUX Prod Mgr
  24. 24. Breaking Down the SILOs QA DBA Sys Adm Net Adm SAN Adm DevUX Prod Mgr Product Team Using Monolithic Delivery Product Team Using Monolithic Delivery
  25. 25. Breaking Down the SILOs QA DBA Sys Adm Net Adm SAN Adm DevUX Prod Mgr Product Team Using Microservices Product Team Using Monolithic Delivery Product Team Using Microservices Product Team Using Microservices Product Team Using Monolithic Delivery
  26. 26. Breaking Down the SILOs QA DBA Sys Adm Net Adm SAN Adm DevUX Prod Mgr Product Team Using Microservices Product Team Using Monolithic Delivery Platform TeamProduct Team Using Microservices Product Team Using Microservices Product Team Using Monolithic Delivery
  27. 27. Breaking Down the SILOs QA DBA Sys Adm Net Adm SAN Adm DevUX Prod Mgr Product Team Using Microservices Product Team Using Monolithic Delivery Platform Team A P I Product Team Using Microservices Product Team Using Microservices Product Team Using Monolithic Delivery
  28. 28. Breaking Down the SILOs QA DBA Sys Adm Net Adm SAN Adm DevUX Prod Mgr Product Team Using Microservices Product Team Using Monolithic Delivery Platform Team DevOps is a Re-Org! A P I Product Team Using Microservices Product Team Using Microservices Product Team Using Monolithic Delivery
  29. 29. Release Plan Developer Developer Developer Developer Developer QA Release Integration Ops Replace Old With New Release Monolithic service updates Works well with a small number of developers and a single language like php, java or ruby
  30. 30. Release Plan Developer Developer Developer Developer Developer QA Release Integration Ops Replace Old With New Release Bugs Monolithic service updates Works well with a small number of developers and a single language like php, java or ruby
  31. 31. Release Plan Developer Developer Developer Developer Developer QA Release Integration Ops Replace Old With New Release Bugs Bugs Monolithic service updates Works well with a small number of developers and a single language like php, java or ruby
  32. 32. Use monolithic apps for small teams, simple systems and when you must, to optimize for efficiency and latency
  33. 33. Developer Developer Developer Developer Developer Old Release Still Running Release Plan Release Plan Release Plan Release Plan Immutable microservice deployment scales, is faster with large teams and diverse platform components
  34. 34. Developer Developer Developer Developer Developer Old Release Still Running Release Plan Release Plan Release Plan Release Plan Deploy Feature to Production Deploy Feature to Production Deploy Feature to Production Deploy Feature to Production Immutable microservice deployment scales, is faster with large teams and diverse platform components
  35. 35. Developer Developer Developer Developer Developer Old Release Still Running Release Plan Release Plan Release Plan Release Plan Deploy Feature to Production Deploy Feature to Production Deploy Feature to Production Deploy Feature to Production Bugs Immutable microservice deployment scales, is faster with large teams and diverse platform components
  36. 36. Developer Developer Developer Developer Developer Old Release Still Running Release Plan Release Plan Release Plan Release Plan Deploy Feature to Production Deploy Feature to Production Deploy Feature to Production Deploy Feature to Production Bugs Deploy Feature to Production Immutable microservice deployment scales, is faster with large teams and diverse platform components
  37. 37. Developer Developer Run What You Wrote Developer Developer
  38. 38. Developer Developer Run What You Wrote Micro service Micro service Micro service Micro service Micro service Micro service Micro service Developer Developer
  39. 39. Developer Developer Run What You Wrote Micro service Micro service Micro service Micro service Micro service Micro service Micro service Developer Developer Monitoring Tools
  40. 40. DeveloperDeveloper Developer Run What You Wrote Micro service Micro service Micro service Micro service Micro service Micro service Micro service Developer Developer Monitoring Tools
  41. 41. DeveloperDeveloper Developer Run What You Wrote Micro service Micro service Micro service Micro service Micro service Micro service Micro service Developer Developer Site Reliability Monitoring Tools Availability Metrics 99.95% customer success rate
  42. 42. DeveloperDeveloper Developer Run What You Wrote Micro service Micro service Micro service Micro service Micro service Micro service Micro service Developer Developer Manager Manager Site Reliability Monitoring Tools Availability Metrics 99.95% customer success rate
  43. 43. DeveloperDeveloper Developer Run What You Wrote Micro service Micro service Micro service Micro service Micro service Micro service Micro service Developer Developer Manager Manager VP Engineering Site Reliability Monitoring Tools Availability Metrics 99.95% customer success rate
  44. 44. Change One Thing at a Time!
  45. 45. What Happened? Rate of change increased Cost and size and risk of change reduced
  46. 46. Low Cost of Change Using Docker Developers • Compile/Build • Seconds Extend container • Package dependencies • Seconds PaaS deploy Container • Docker startup • Seconds
  47. 47. Low Cost of Change Using Docker Fast tooling supports continuous delivery of many tiny changes Developers • Compile/Build • Seconds Extend container • Package dependencies • Seconds PaaS deploy Container • Docker startup • Seconds
  48. 48. Disruptor: Continuous Delivery with Containerized Microservices
  49. 49. Microservices
  50. 50. A Microservice Definition Loosely coupled service oriented architecture with bounded contexts
  51. 51. A Microservice Definition Loosely coupled service oriented architecture with bounded contexts If every service has to be updated at the same time it’s not loosely coupled
  52. 52. A Microservice Definition Loosely coupled service oriented architecture with bounded contexts If every service has to be updated at the same time it’s not loosely coupled If you have to know too much about surrounding services you don’t have a bounded context. See the Domain Driven Design book by Eric Evans.
  53. 53. Coupling Concerns http://en.wikipedia.org/wiki/Conway's_law ●Conway’s Law - organizational coupling ●Centralized Database Schemas ●Enterprise Service Bus - centralized message queues ●Inflexible Protocol Versioning
  54. 54. Speeding Up Deployments Datacenter Snowflakes • Deploy in months • Live for years
  55. 55. Speeding Up Deployments Datacenter Snowflakes • Deploy in months • Live for years Virtualized and Cloud • Deploy in minutes • Live for weeks
  56. 56. Speeding Up Deployments Datacenter Snowflakes • Deploy in months • Live for years Virtualized and Cloud • Deploy in minutes • Live for weeks Container Deployments • Deploy in seconds • Live for minutes/hours
  57. 57. Speeding Up Deployments Datacenter Snowflakes • Deploy in months • Live for years Virtualized and Cloud • Deploy in minutes • Live for weeks Container Deployments • Deploy in seconds • Live for minutes/hours AWS Lambda Events • Respond in milliseconds • Live for seconds
  58. 58. Speeding Up Deployments Measuring CPU usage once a minute makes no sense for containers… Coping with rate of change is the first challenge for monitoring tools. Datacenter Snowflakes • Deploy in months • Live for years Virtualized and Cloud • Deploy in minutes • Live for weeks Container Deployments • Deploy in seconds • Live for minutes/hours AWS Lambda Events • Respond in milliseconds • Live for seconds
  59. 59. Inspiration
  60. 60. Challenges for Microservice Platforms
  61. 61. Managing Scale
  62. 62. A Possible Hierarchy Continents Regions Zones Services Versions Containers Instances How Many? 3 to 5 2-4 per Continent 1-5 per Region 100’s per Zone Many per Service 1000’s per Version 10,000’s It’s much more challenging than just a large number of machines
  63. 63. Flow
  64. 64. Some tools can show the request flow across a few services
  65. 65. But interesting architectures have a lot of microservices! Flow visualization is a big challenge. See http://www.slideshare.net/LappleApple/gilt-from-monolith-ruby-app-to-micro-service-scala-service-architecture
  66. 66. Failures
  67. 67. ELB Load Balancer Zuul API Proxy Karyon Business Logic Staash Data Access Layer Priam Cassandra Datastore Simple NetflixOSS style microservices architecture on three AWS Availability Zones
  68. 68. ELB Load Balancer Zuul API Proxy Karyon Business Logic Staash Data Access Layer Priam Cassandra Datastore Simple NetflixOSS style microservices architecture on three AWS Availability Zones
  69. 69. ELB Load Balancer Zuul API Proxy Karyon Business Logic Staash Data Access Layer Priam Cassandra Datastore Simple NetflixOSS style microservices architecture on three AWS Availability Zones Zone partition/failure What should you do? What should monitors show?
  70. 70. ELB Load Balancer Zuul API Proxy Karyon Business Logic Staash Data Access Layer Priam Cassandra Datastore Simple NetflixOSS style microservices architecture on three AWS Availability Zones Zone partition/failure What should you do? What should monitors show? By design, everything works with 2 of 3 zones running. This is not an outage, inform but don’t touch anything! Halt deployments perhaps?
  71. 71. ELB Load Balancer Zuul API Proxy Karyon Business Logic Staash Data Access Layer Priam Cassandra Datastore Simple NetflixOSS style microservices architecture on three AWS Availability Zones Zone partition/failure What should you do? What should monitors show? By design, everything works with 2 of 3 zones running. This is not an outage, inform but don’t touch anything! Halt deployments perhaps? Challenge: understand and communicate common microservice failure patterns.
  72. 72. Testing
  73. 73. Testing monitoring tools at scale gets expensive quickly…
  74. 74. Simulation
  75. 75. Simulated Microservices Model and visualize microservices Simulate interesting architectures Generate large scale configurations Eventually stress test real tools See github.com/adrianco/spigo Simulate Protocol Interactions in Go Visualize with D3 ELB Load Balancer Zuul API Proxy Karyon Business Logic Staash Data Access Layer Priam Cassandra Datastore Three Availability Zones
  76. 76. Definition of an architecture { "arch": "netflixoss", "description":"A very simple Netflix service. See http://netflix.github.io/ to decode the package names", "version": "arch-0.0", "victim": "homepage", "services": [ { "name": "cassSubscriber", "package": "priamCassandra", "count": 6, "regions": 1, "dependencies": ["cassSubscriber", "eureka"]}, { "name": "evcacheSubscriber","package": "store", "count": 3, "regions": 1, "dependencies": []}, { "name": "subscriber", "package": "staash", "count": 6, "regions": 1, "dependencies": ["cassSubscriber", "evcacheSubscriber"]} { "name": "login", "package": "karyon", "count": 18, "regions": 1, "dependencies": ["subscriber"]}, { "name": "homepage", "package": "karyon", "count": 24, "regions": 1, "dependencies": ["subscriber"]}, { "name": "wwwproxy", "package": "zuul", "count": 6, "regions": 1, "dependencies": ["login", "homepage"]}, { "name": "www-elb", "package": "elb", "count": 0, "regions": 1, "dependencies": ["wwwproxy"]}, { "name": "www", "package": "denominator", "count": 0, "regions": 0, "dependencies": ["www-elb"]} ] } Header includes chaos monkey victim New tier name Tier package Region count: 1 Node count List of tier dependencies
  77. 77. Why Build Spigo/Simianviz? Generate test microservice configurations at scale Stress monitoring tools display capabilities Eventually (i.e. not implemented yet) Dynamically vary configuration: autoscale, code push Chaos monkey for microservice, zone, region failures D3 websocket dynamic browser interface
  78. 78. Simulated Game Day Testing and Training…
  79. 79. Conference Driven Development… OSCON Microservices Workshop
  80. 80. What’s Next?
  81. 81. Developer Concerns Agile, Lean & Rugged (Faster, Cheaper and Safer) See GOTO London Sept 2015
  82. 82. Forward Thinking
  83. 83. Forward Thinking
  84. 84. Any Questions? Disclosure: some of the companies mentioned may be Battery Ventures Portfolio Companies See www.battery.com for a list of portfolio investments ● Battery Ventures http://www.battery.com ● Adrian’s Tweets @adrianco and Blog http://perfcap.blogspot.com ● Slideshare http://slideshare.com/adriancockcroft ● Monitorama Opening Keynote Portland OR - May 7 th , 2014 ● GOTO Chicago Opening Keynote May 20 th , 2014 ● Qcon New York – Speed and Scale - June 11 th , 2014 ● Structure - Cloud Trends - San Francisco - June 19th, 2014 ● GOTO Copenhagen/Aarhus – Fast Delivery - Denmark – Sept 25 th , 2014 ● DevOps Enterprise Summit - San Francisco - Oct 21-23rd, 2014 #DOES14 ● GOTO Berlin - Migrating to Microservices - Germany - Nov 6th, 2014 ● AWS Re:Invent - Cloud Native Cost Optimization - Las Vegas - November 14th, 2014 ● O’Reilly Software Architecture Conference - Fast Delivery, Monitoring Challenge - Boston March 16th 2015
  85. 85. Security Visit http://www.battery.com/our-companies/ for a full list of all portfolio companies in which all Battery Funds have invested. Palo Alto Networks Enterprise IT Operations & Management Big DataCompute Networking Storage

×