Architecting for Continuous Delivery
Die SlideShare-Präsentation wird heruntergeladen.
×
Hier ansehen
Empfohlen
Weitere Verwandte Inhalte
Diashows für Sie (20)
Ähnlich wie Cloud Journey: Lifting a Major Product to Kubernetes (20)
Weitere von Haufe-Lexware GmbH & Co KG (20)
Aktuellste (20)
Cloud Journey: Lifting a Major Product to Kubernetes
- 1. Welcome to & Dock8s Meetup Robert Werlich Site Reliability Engineer robert.werlich@verivox.com Marlen Blaube Senior HR Business Partner marlen.blaube@verivox.com
- 2. Cloud Journey: Lifting a Major Product to Kubernetes Dock8s Meetup Heidelberg Feb 27th 2019 Martin Danielsson, Haufe Group, Freiburg @donmartin76 (Twitter, Github)
- 3. Dock8s Meetup Heidelberg, February 27th 2019 whoami C:> WINDOWS.EXE C/C++/C# Background 10+ years $ docker ps Containers & Kubernetes Since ~4 years wicked.haufe.io maintainer OSS API Management Solution Architect Developer since 2006
- 4. Dock8s Meetup Heidelberg, February 27th 2019
- 5. Dock8s Meetup Heidelberg, February 27th 2019 Agenda Operations
- 6. Planning & Politics We‘ll set the scene a little.
- 7. Dock8s Meetup Heidelberg, February 27th 2019 Some numbers 100+ active git repos 874k LOC 10-15 Developers 200-500 concur- rent users Typically 100 req/s 448 GB RAM 56 Cores
- 8. Major revenue Strategic move to containers Modular Architecture Without Container Experience Hosted with Hoster (€€€) Long Release cycles (LOTS of) Manual Work for Releases Little Operations Insight Error tracking very difficult Non-Parity Dev/Test/Prod (Cost!) Legacy Web App (Java based)
- 9. Dock8s Meetup Heidelberg, February 27th 2019 Vision – Goals Enabling CI/CD Automatic Provisioning Full Insight Minimize Ops
- 10. Dock8s Meetup Heidelberg, February 27th 2019 Let‘s go DevOps in the Cloud!
- 11. Dock8s Meetup Heidelberg, February 27th 2019 Project Interfaces Technology Processes HR Topics (Operations) Stakeholder Management
- 12. Dock8s Meetup Heidelberg, February 27th 2019 Stakeholder Management CONVINCE THEM, DON‘T PERSUADE THEM COMMUNICATE OFTEN AND CLEARLY DON‘T UNDERESTIMATE TASKS AT HAND BE TRANSPARENT SHARE SUCCESSES BUT ALSO FAILURES!
- 13. Dock8s Meetup Heidelberg, February 27th 2019 Team Setup – Vision 100% DevOps Engineers T-Shaped Engineers No dedicated manual testers Automate! YBI, YRI. Ops experience?
- 14. Dock8s Meetup Heidelberg, February 27th 2019 Some HR topics Release Managers? Operations Responsibility? Quality Engineers (testers)? On Call Duty?
- 15. Technology This is what you came for…?
- 16. Dock8s Meetup Heidelberg, February 27th 2019 Technology Stack Kubernetes Azure (Public Cloud)
- 17. Dock8s Meetup Heidelberg, February 27th 2019 Steps to DevOps Happiness Provision Deploy CI/CD Weekly for Production, Daily for Dev/Test Ship when ready!
- 18. Dock8s Meetup Heidelberg, February 27th 2019 Wait, uh, what…? Target “No-Ops” No long-running systems Enable validation of 3rd Party component upgrades Incremental changes Practice Disaster Recovery Daily 100% Reproducible Deployments On-demand Production Identical Environments
- 19. Dock8s Meetup Heidelberg, February 27th 2019 Code & Pipelines So, it‘s all… … and pipelines are also code
- 20. Dock8s Meetup Heidelberg, February 27th 2019 Incremental Backend Development Merge feature to master •After code review •Including test suite changes Build master branch •Includes unit testing •First integration tests Deploy to integration system •Blue/Green with integration tests Deploy to Production •Blue/Green with integration tests
- 21. Dock8s Meetup Heidelberg, February 27th 2019 Incremental Frontend Development Merge feature to master •After code review •Including test suite changes Build master branch •Includes unit testing •First integration tests Deploy to integration system •Run e2e integration tests •Rollback if failing Deploy to Production •Run e2e integration tests •Rollback if failing
- 22. Dock8s Meetup Heidelberg, February 27th 2019 Stateless Components Stateful Components
- 23. Dock8s Meetup Heidelberg, February 27th 2019 Full Provisioning Create backup Provision new infrastructure •From backups •Same as disaster recovery! Deploy components •Using deployment pipelines •Partly parallelized Top level DNS switch •Using DNS traffic manager Destroy old infrastructure •If tests succeed
- 24. Dock8s Meetup Heidelberg, February 27th 2019 Persistence Options Roll your own persistence Persistence “as a service” Self managed VMs (incl. NFS) Managed Disks (AWS EBS, Azure Managed Disks) DBaaS (many options) Files as a service (AWS EFS, Azure Files) Gluster/Ceph FS (cluster)
- 25. Dock8s Meetup Heidelberg, February 27th 2019 iDesk2 Deployment Architecture Resource Group Kubernetes Cluster ks8 Master ks8 Agent ks8 Agent n … NFS VM(s) Postgres VM(s) Disks Disks • Azure Files not fast enough • Legacy components depend on UNIX rights (Azure Files is SMB) • Azure Disks only ReadWriteOnce • Azure PGaaS was not yet available • More „bang for your buck“ • PG Admin knowledge in Team
- 26. Dock8s Meetup Heidelberg, February 27th 2019 Endless Variants
- 27. Dock8s Meetup Heidelberg, February 27th 2019 Some hints… Assess your Persistence Needs early on If possible, use DBaaS (avoid NIH syndrome) Externalize Configuration Shared File Storage is not “Cloud Native”
- 28. Operations No. You don‘t get around it. Sorry.
- 29. Dock8s Meetup Heidelberg, February 27th 2019 Now that we have Kubernetes…? Self healing Robust Production Ready Battle proven “Vertrauen ist gut... … Kontrolle ist besser!” Complex Additional Abstraction Layer
- 30. Dock8s Meetup Heidelberg, February 27th 2019 “Kontrolle” - What do you mean? Detecting these things is a start...
- 31. Dock8s Meetup Heidelberg, February 27th 2019 Fail: Lyin’ Monitors End-to-End Monitoring ALL GOOD People logging in 500 … an entire weekend.
- 32. Dock8s Meetup Heidelberg, February 27th 2019 Instrument Monitor and Alert Enable Insight
- 33. Dock8s Meetup Heidelberg, February 27th 2019 Prometheu s A Metrics Endpoint http://A:8080/metrics JVM Metrics Node.js Metrics VM Exporters (node_exporter) DB Exporters (pg_exporter) Kubernetes Statistics Prometheus Client based Custom Exporters ... BTime Series DB
- 34. Dock8s Meetup Heidelberg, February 27th 2019 Alertmanage r
- 35. Dock8s Meetup Heidelberg, February 27th 2019 Metrics White Box Black Box Counters GaugesHistograms Summaries Application Network Latencies Errors Timeouts Infrastructure Disk Space CPU Memory Pod Status
- 36. Dock8s Meetup Heidelberg, February 27th 2019 Friday 9 o’clock Newsletter
- 37. Dock8s Meetup Heidelberg, February 27th 2019 205’886
- 38. Dock8s Meetup Heidelberg, February 27th 2019 Alerting? On what?
- 39. Dock8s Meetup Heidelberg, February 27th 2019 Availability Infrastructure
- 40. https://www.zazzle.com/nines_dont_matter_t_shirt-235118578582589495 Charity Majors says… (@mipsytipsy)
- 41. Dock8s Meetup Heidelberg, February 27th 2019 Percentage of document retrieval requests served within 0.25 and 1s Percentage of search requests answered within 1, 3 and 7.5s Percentage of Error Pages Indicators 95% and 98.5% 50%, 95% and 98.5% <1% Agreements Service Level
- 42. Dock8s Meetup Heidelberg, February 27th 2019
- 43. Dock8s Meetup Heidelberg, February 27th 2019 Holistic View Instrument early (and lots) Deployments easier Less fear of change We are in control! hope and think we
- 44. Dock8s Meetup Heidelberg, February 27th 2019 Fails: Resiliency Issues VMs are sometimes patched and restarted. Or they just die. So will any service on them. Networks are unreliable. Connections will fail. Use (libraries for) circuit breakers and retries. Re-establishing TLS on each call to external services is expensive. … and the service will hate you. Use Keep-Alive. SPOFs will eventually fail. Assess and act. Learn how to detect problems.
- 45. Conclusion Was it worth it?
- 46. Dock8s Meetup Heidelberg, February 27th 2019 Would we do it again?
- 47. Dock8s Meetup Heidelberg, February 27th 2019 Key Performance Indicators • >70% Cost Saving • Release Effort down >98% via automation • Higher Release Pace (3-5/y to 15-20/mo) • Performance measurable • Faster Reaction to Issues • Unlocks Cloud Technology Dock8s Meetup Heidelberg, February 27th 2019
- 48. Dock8s Meetup Heidelberg, February 27th 2019 k8s Ops possible as a Team Requires full automation (also test) Team dedication Rethinking ops is challenging No Silver Bullet Assess your requirements
- 49. Dock8s Meetup Heidelberg, February 27th 2019 Some Links… kubernetes.io prometheus.io grafana.com azure.com aws.amazon.com Twitter @donmartin76 GitHub donmartin76 We’re hiring! www.haufegroup.com/en/career
