Hashicorp Nomad and Lyft Envoy have been brought together as fundamental building blocks for Nelson, a new open source project from Verizon Labs that enables blazing fast continuous delivery, fully automated lifecycle and cleanup for applications, along with dynamic runtime traffic management. Automatically shift your production traffic back and forth between different versions of your immutable service infrastructure without any outages. Encrypt all your traffic transparently and provide best-in-class security without your engineering staff having to learn how your infrastructure and security sausages are made.
3. Problem
• Provisioning applications is still too slow (bare metal or cloud).
• Runtime traffic control systems are medieval at best.
• Coupling CI and CD creates monolithic operational systems.
• These systems do everything. This is a distinct problem.
• Current market solutions limited or hard to adopt.
• Most teams have brittle, painful automation nobody wants to use.
• Many teams attempt CD ignorant of the side-effects.
4. Lessons
• Automate every part of the system.
• Testing a distributed system locally is a fable.
• Emergent properties. Scaling issues etc.
• Uniformity is highly desirable and wildly advantageous.
• Beautiful, unique snowflakes are however, inevitable.
• Automated lifecycle management is required.
5. Goals
• Use the minimally powerful components.
• System elements should be awesome at just one thing.
• Reduce overall platform complexity.
• Increase responsibility of engineering teams. Break it, you bought it.
• Decentralize process gatekeepers.
• No build team. No ticket filing for deployments. No configuration
management.
6. Goals
• All application specifications are checked in.
• Build. Deployment. Alerting etc.
• Reduce deployment time 2 minutes or less.
• Support multi-DC topologies from the get-go.
• Automatic credential management and secure-introduction
• Transparent, strong encryption for application I/O on the wire.
9. Nomad
• Use a farm of servers as a single resource pool: RAM, CPU, etc
• Typically used at larger scale, becoming more common.
• Blazing fast: only placement without provisioning.
• Integration with Vault, so secure-introduction works OOTB.
• Monolithic resource manager & scheduler [1]
• Several open-source & commercial alternatives: Mesos, k8s etc
[1] https://research.google.com/pubs/pub43438.html
11. Envoy
• Fast L4 and L7 proxy solving many practical ops concerns.
• Open-sourced end of 2016; blossomed since.
• Lyft, Google, IBM et al all actively contributing.
• Make applications dumb; invest in a single element of routing infra
• Retries, Circuit Breaking, TLS Encryption etc
• Integrate horizontally, not vertically
• Integrate with whatever discovery system you want via APIs.
15. Overview
• Github driven developer workflow (.com or enterprise).
• Choose whatever build / CI system you want.
• State of the art runtime routing via Envoy.
• Secure introduction for safe distribution of credentials from Vault.
• Integrated with Nomad; target any datacenter running a scheduler.
• Integrated alert definition with Prometheus.
41. Discovery.
• Discovery protocol written to Consul KV for every stack
• We call this Lighthouse protocol
• Application dependencies are declared a-priori.
• You cannot route to that which you do not tell Nelson about.
• Makes for awesome auditing and security.
• Language implementations need only consume the protocol.
42. Routing.
• Non-prescriptive approach to routing tier implementation.
• Provides a control plane protocol describe routing actions.
• Typically implemented with Envoy, but you can choose.
• Minor application changes required.
• Incentivized these with tracing and context propagation.
• Models traffic shifting as a time vs traffic policy curve.
47. Challenges
• Non-trivial level of investment and execution.
• Tight integration with Hashistack is both pro or con.
• Containerizing legacy applications can be “interesting”.
• Migration can be a challenge if not collocated with “the new world”.
• Small organizations better served by existing solutions.
48. Future Work
• Aim to open-source supporting and complimentary tools.
• Consul / Envoy integration. Cost analysis subsystem.
• Make Nelson easier to extend for third-parties
• eDSL for workflows, externalize policy algebra
• General “plugin” system is a possibility
• Listen to the community feedback.
49. Summary
• Fully automated application lifecycle: no manual housekeeping.
• Choose whatever CI setup best fits your team.
• Secure your deployments.
• Transparent mTLS and rotating credentials.
• Automatic Vault policy management.
• Provide rigor to your application Death Star.