As a company, Verizon networks and infrastructure touch nearly 70% of global internet traffic every single day. The many datacenters that support this - and many other large-scale Verizon services - are our lifeblood. This talk provides a glimpse into the work being done to reimagine the way in which we design and operate the software that runs our internal computing grids, and how we enable a large body of development staff to ship jobs and services to the grid every single day. We’ll cover how Consul and Vault make for invaluable building blocks in modern distributed systems, and highlight the importance of empowering teams through well designed infrastructure systems.

1. LARGE-SCALE
INFRASTRUCTURE AUTOMATION
Timothy Perrett
Hashiconf 2016

2. HELLO.

3. STATE OF THE UNION
INDUSTRY
^

4. PROLIFERATION OF
RESOURCE MANAGEMENT

5. LINUX CONTAINERS
ARE FINALLY POPULAR

6. NOSQL EVERYWHERE

7. NOSQL EVERYWHERE
with RDMS
still commonplace
^

10. OBSERVATIONS.

11. “Infrastructure engineering is 60% social, and
only 40% technical. Changing people is far
more important than changing technology.

12. “Enable sociological change. Technological
changes are an implementation detail.

13. “Operational complexity is often proportional
to the lack of developer responsibility.

14. FLEXIBILITY
CONSTRAINT
Business
Staff

15. FLEXIBILITY
CONSTRAINT
Engineering
Staff

16. “Constraints liberate. Liberties constrain.
- Runar Bjarnason

17. BRIEF HISTORY
We have to go back!

18. 4 YEARS AGO.

19. 4 YEARS AGO.
Pretty typical configuration management.
Centralized Chef servers.
Lots of unmaintainable Ruby.
Ruby that generates Ruby which is
evaluated at runtime (yikes!).
Developer contract is non-existent.
Operations need to understand every
application in detail.
Code complete to finally deployed
took around two weeks.

20. 3 YEARS AGO.

21. 3 YEARS AGO.
Implemented immutable machine images
with Hashicorp Packer.
Developer / Ops contract becomes an RPM/
DEB file along with two YAML manifests.
One manifest for provisioning.
Another for runtime deployment setup.
Drive the entire release workflow from
source repositories.
Orchestrated with many linked Jenkins jobs
and schedules.
Code complete to finally deployed took
around 40 minutes.

22. TODAY.

23. TODAY.
Developer / operations contract is just a
linux container.
Repository contains a YAML manifest.
Realization that placement and
orchestration are entirely separate.
Intelligent and fully automated cleanup.
Application dependency management.
Automated traffic bleeding.
Integrated alerting with prometheus,
general notifications with slack or email.
Code complete to deployed takes
around 5 minutes.

24. NELSON.

25. “Desperate affairs require desperate remedies.
-Vice Admiral Horatio Nelson, 1758-1805

26. GOALS.

27. GOALS.
System elements should be awesome at
just one thing.
Reduce system complexity by increasing
responsibility of engineering teams.
Break it, you bought it.
All application specifications are
checked into source control.
Focus on orchestration, not placement.
Force automation in every aspect of work
Manual access to systems are a crutch
that enables automation avoidance.

29. UNITS.

30. job

31. - name: hello world
type: job
description: >
mindlessly prints hello
world to the console for
five minutes
schedule: hourly
retries: 2
expiration_policy: >
retain-latest-two-major
dependencies:
- ref: example@3.1
unit type
job stuff

32. service

33. - name: howdy
type: service
description: >
always responds with
hello world
ports:
- default->8080/http
expiration_policy: >
retain-latest-two-major
dependencies:
- ref: foobar@3.1
unit type
service stuff

34. edge
proxy

35. - name: foobar-proxy
type: proxy
description: >
proxy inbound from outside
routes:
- name: expose the ssl port
expose: inbound->443/https
destination: edge@1.3->default
expiration_policy: >
retain-until-deprecated
routes

36. WORKFLOWS.

38. failure
domain

39. container
replication

40. alerting
routing
discovery

41. scheduling

42. credentials

43. LIFECYCLE.

44. User
activated

45. Graph
Pruning

46. Upgraded!

47. Automatically
Terminated
X
X

48. LIFECYCLE.
Various cleanup strategies
Graph pruning
Explicit deprecation cycles
User selected policies for versions
Retain last two major
Retain last two minor
Retain latest
Retain always
Eliminates the “Do we still need this?”
conversations between ops and
development.
- name: hello world
type: job
description: >
mindlessly prints hello
world to the console for
five minutes
schedule: hourly
retries: 2
expiration_policy: >
retain-latest-two-major
dependencies:
- ref: example@3.1

49. TL;DR.
Automate everything. Your future sanity depends on it.
Define concrete protocols at system integration points; favor machine verifiable
protocols where possible.
Your path to success involves people. Listen, learn and be open for criticism.
Consul & Vault provide building-block functionality that just works.
Never settle for mediocre tools.
Know when buying is better than building, but don’t be afraid to build
if it adds value.

50. EOF
WE’RE
HIRING!
timperrett
github.com/verizon