Slides from my talk at Kubernetes Meetup (Feb 15, 2017)

1. From CoreOS
To Kubernetes
Denis Izmaylov
Feb 16, 2017

2. Denis Izmaylov
1. Over 17 years of Software and Web
development experience
2. The last 7 years focused exclusively on
Front-end and Web: 15+ projects,
including React.js, SPA and high-load
3. Open Source contributor
4. Talks: HighLoad++ 2015, AgileDays 2016,
DevConf 2016, React Amsterdam, etc
5. Regular speaker on MoscowJS
6. Few popular articles and interviews
on Medium and Habrahabr
7. Events: Moscow Node.js Meetup, React
Moscow Meetup and this one

3. Web and Mobile development services

4. 1. We develop Web, Mobile
and DevOps projects for
our customers
2. Global international
market (EU, APAC, US, HK)
3. Most efficient and
performant technologies
4. More than 20 talents
engineers and designers
5. Remote-friendly

5. Prerequisite Knowledge
1. Linux
2. Docker basic
3. Kubernetes basic
4. Monolithic architecture
5. Microservices principles
6. YAML

6. Chapter 2

7. Watch the previous talk:
Moscow Node.js Meetup 8
at Yandex (2016, July 13)
https://events.yandex.ru/events/yagosti/13-jul-2016/

8. What problems can you get
with CoreOS?

9. Why Kubernetes is
not enough?

10. How to scale fast
and effortless?

11. 11
Customer Goals
1. Production cluster on 3-5 machines
2. Be ready for fast growing (up to 10x)
3. Fault tolerance
4. Faster changes delivery
5. Full control for each service (in-house,
independent, no-SaaS)
6. Blue-Green (Zero Downtime) Deployment
7. Good DX (Developer Experience)

12. 12
Solution
1. Microservices
2. Blue-Green (Zero Downtime)
Deployment
3. Scale Cube
4. CoreOS

13. Microservices

14. 14
Monolithic Architecture

15. 15
Problems
1. Slow
2. Expensive
3. Hard
4. High risks
5. Vendor lock-in
6. Low quality

16. Typical Monolith
16

17. 17
What if…?
Each function
with dependencies

18. 18
What if…?
Each function
with dependencies

19. 19
What if…?

20. 20
Microservices Architecture

21. Microservices Architecture
Single Responsibility Principle
21

22. 22
Problems
1. Slow
2. Expensive
3. Hard
4. High risks
5. Vendor lock-in
6. Low quality
1. Fast
2. Cheap
3. Easy
4. Low risks
5. Vendor-less
6. Cost/quality
Solved

23. 23
Effects
1. Resistance
• Fail-over
• Isolated failures
2. Delivery speed
3. Changes are simple
4. Freedom
5. Security

24. www.reactivemanifesto.org

25. 25
“Microservices allow engineering teams
to move quickly to grow a product…
assuming they don’t get bogged down
by the complexity of operating a
distributed system”

26. 26
Solution
1. Microservices
2. Blue-Green (Zero Downtime)
Deployment
3. Scale Cube
4. CoreOS

27. 27
Blue-Green Deployment

28. 28
Solution
1. Microservices
2. Blue-Green (Zero Downtime)
Deployment
3. Scale Cube
4. CoreOS

29. Scale Cube

30. 30
Scale Cube
• X-axis: by cloning Services
• Y-axis: by functional decomposition
• Z-axis: by sharding databases

31. 31
Solution
1. Microservices
2. Blue-Green (Zero Downtime)
Deployment
3. Scale Cube
4. CoreOS

32. 32

33. 33
1. First Container-optimized OS: 3+ years, since July 2013
2. Mini OS: based on ChromeOS, small and fast
3. Security-focused: by using auto-updates (like
browsers)
4. No package manager: each application is running as
a Container on top of OS
5. Built-in support: Containers, Container networking,
Service discovery
6. Services: runs through systemd
7. Configuration: one Cloud Config file

34. 34
Components
1. etcd
2. fleet
3. flannel
* state on March, 2016

35. 35
etcd
Distributed key-value storage:
1. Common
configuration
2. Service
Discovery
3. Resource
locking

36. 36
fleet
Distributed systemd

37. 37
flannel
Overlay network for your containers

38. 38

39. 39
Cloud Config

40. 40
test-app@.service

41. 41
Deploy & Destroy

42. 42
Summary
1. CoreOS and Docker: you don’t think
about OS anymore
2. Containers space
3. Now a server is just a core of your“CPU”
4. Auto-schedule: you don’t have to know
where is your Container is running
5. Configuration-less
6. Amazing expandable

43. Breaking News

45. 45
Dec 12, 2016
https://coreos.com/blog/tectonic-self-
driving.html#coreos-linux-is-now-container-
linux

46. Moving from fleet
to Kubernetes

48. Today

51. 51
Solution
1. Microservices
2. Blue-Green (Zero Downtime)
Deployment
3. Scale Cube
4. CoreOS

52. 52
Customer Goals
Production cluster on 3-5 machines
Be ready for fast growing (up to 10x)
Fault tolerance
Faster changes delivery
Full control for each service (in-house,
independent, no-SaaS)
Blue-Green (Zero Downtime) Deployment
Good DX (Developer Experience)

54. Wake up!

55. 55
Real Life
1. Launch a new containers
2. Integrate containers between each
other
3. Connect containers with Load Balancers
4. Rolling updates for the launched
containers
5. Use Blue-Green Deployment strategy

56. 56
Simple Steps
1. Create Fleet unit file for each service
2. Each service requires Service Discovery
3. Sidekick model: create a Discovery Unit
4. Rolling updates: small set of commands
5. BGD deployment: small script

57. Dive Deeper

58. 58
Project Profile
1. Isomorphic React.js Application
2. RESTful API on Node.js
3. Serve Static service (nginx)
4. Golang service
5. C++ service
6. PHP service
7. JSON Configuration service

59. 59
Go Ahead
• Fleet unit for each service
• Discovery Unit for each service
• X-scaling: multiple instances
• Environments (dev, stage, prod)

60. 60
Go Ahead
• Fleet unit for each service
• Discovery Unit for each service
• X-scaling: multiple instances
• Environments (dev, stage, prod)
6
2
6
3

61. 61
Go Ahead
6
2
6
3
x
x
x

62. 62
Go Ahead
6
2
6
3
x
x
x
216 containers

63. 216 containers

64. Very interesting

65. I can’t configure CoreOS anymore

66. 66
Real Summary
1. CoreOS is OS only
2. Not enough for microservices
management
3. Good enough as a platform of other
tools (PaaS, SaaS, etc)
4. Configure CoreOS Cluster Step-by-Step:
https://github.com/axept/devops-open

67. From CoreOS

68. 68
Next Level Requirements
1. Microservices Management
2. Continuous Delivery / Deployment
3. SSL support
4. TLD support
5. Monitoring
6. Alerting and Notifications

69. May 2016

71. 71
Kubernetes
1. Open-Source Cluster Management
2. Originally developed by Google
3. Open-sourced in June 2014
4. Deploy and manage applications
declaratively
5. Declarative: state your desired results, let
the system actuate
6. Control loops: observe, rectify, repeat
7. No groups: labels is more powerful

72. Kubernetes is greek for“helmsmen”

73. – How can we spend more time
building useful services
and less time maintaining the platform?

74. 74
Some Concepts
1. Controller: drives current state to target
state
2. Pod: a small group of tightly-integrated
Containers
3. Label: identifying metadata attached to
other objects (e.g. to Pods)
4. Service: exposes a name, port and IP for a
groups of Pods
5. Ingress: connects Load Balancers with
Services by using URL rules

75. 75
Controller
1. Run on Master
continuously
2. Each K8s object gets
its own Controller
3. Control Loop:
• get state
• calculate diff
• apply changes
4. Pluggable and lightweight
Get state
from etcd
Compare
with
worker’s
state
Actuate it

76. 76
Pods
1. Group of Containers
2. One or multiply containers
3. Has its own IP and localhost
4. Scale together
5. Shared secrets
6. Shared volumes
7. Shared resources (CPU and RAM)
8. Can be tagged with labels

77. 77
Label
1. Identifying metadata
2. Can be attached to any API
object
3. Generally represents identity
4. It could be application name,
role, phase, environment,
version, etc
5. Using by Selectors
6. The grouping mechanism only

78. 78
Example for Labels
app: demo
phase: dev
role: FE
app: demo
phase: dev
role: BE
app: demo
phase: test
role: FE
app: demo
phase: test
role: BE

79. 79
Selectors
Think about it like SQL:“select … where …”
app: demo
phase: dev
role: FE
app: demo
phase: dev
role: BE
app: demo
phase: test
role: FE
app: demo
phase: test
role: BE
app = demo

80. 80
Selectors
Think about it like SQL:“select … where …”
app: demo
phase: dev
role: FE
app: demo
phase: dev
role: BE
app: demo
phase: test
role: FE
app: demo
phase: test
role: BE
app=demo
role=FE

81. 81
Selectors
Think about it like SQL:“select … where …”
app: demo
phase: dev
role: FE
app: demo
phase: dev
role: BE
app: demo
phase: test
role: FE
app: demo
phase: test
role: BE
app=demo
role=BE

82. 82
Selectors
Think about it like SQL:“select … where …”
app: demo
phase: dev
role: FE
app: demo
phase: dev
role: BE
app: demo
phase: test
role: FE
app: demo
phase: test
role: BE
app = demo
phase = dev

83. 83
Selectors
Think about it like SQL:“select … where …”
app: demo
phase: dev
role: FE
app: demo
phase: dev
role: BE
app: demo
phase: test
role: FE
app: demo
phase: test
role: BE
app = demo
phase = test

84. 84
Services
1. Serve as a kind of
Load Balancers for Pods
2. Expose name, port and
stable IP for a group of Pods
3. IP can be internal or public
4. Can implement Service Discovery
between applications and features
to support Zero-downtime deployments

85. 85
Ingress
1. Works with Load
Balancers, Cloud LB,
nginx, etc
2. Provides a single root
URL to multiple services
3. Publicly expose private
networks
4. Can use HTTP Headers
5. Supports SSL

86. And a lot of other things

87. Explore
Feature Tracking and Backlog:
https://github.com/kubernetes/features

88. 88
In next talk
1. How to configure Concourse CI
with Kubernetes properly?
2. How to implement TLD (Top-Level Domains,
e.g. google.de, yahoo.eu, yandex.ru)?
3. How to configure persistent SSL certificates
and temporary SSL certificates by
Letsencrypt?
4. How to configure monitoring and alerting
using Grafana?
5. How to install the cluster in a one click?

89. In next talk

90. 90
Summary
1. Kubernetes saved us from“1000 Fleet Units”
hell in CoreOS and a decades of custom
scripts
2. K8s saved us a lot of months
3. Some other interesting features:
• Role-based Access Control
• AWS, GCE, Azure support
• Apache License Version 2.0

91. 91
What We Got
1. Micro-services Management
2. Continuous Delivery / Deployment
3. Easy SSL and TLD management
4. Monitoring
5. Notifications

92. 92
Achievements
1. Production cluster on 3-5 machines
2. Be ready for fast growing (up to 10x)
3. Fault tolerance
4. Faster changes delivery
5. Full control for each service (in-house,
independent, no-SaaS)
6. Blue-Green (Zero Downtime) Deployment
7. Great DX (Developer Experience)

93. That’s not enough!

94. We developed
One-Click Installer

95. 95
One-Click Installer
1. Uses Ansible
2. Prepare Configuration directory
with keys, vars, secrets, etc
3. Optional - prepare network interfaces on target
machines
4. Configure servers list and SSH credentials
5. Launch the script
6. Get a new cluster in 3 minutes with
isolated CI and Private Docker Registry

96. 96
Effects
1. CoreOS, Kubernetes and Application became
an isolated node
2. Micro-clusters Architecture

98. 98
Effects
1. CoreOS, Kubernetes and Application became
an isolated node
2. Micro-clusters Architecture
3. Possible to get Cross-cluster Load Balancing and
Service Discovery
4. Automated Installation
5. 24/7 Support friendly
6. No“Black box”inside
7. Efficient Geo-scalability

99. 99
Take aways
1. CoreOS is not about microservices management
2. CoreOS is just Container OS (Container Linux!)
3. CoreOS is good as a tool for SaaS, PaaS, etc
4. Kubernetes is really great for microservices
management and large clusters
5. Kubernetes is the new operation system (OS)
6. Some interesting things will be in next talk

100. “Microservices allow engineering teams
to move quickly to grow a product…
assuming they don’t get bogged down
by the complexity of operating a
distributed system”
7.

101. 101
Read (1 of 2)
1. Kubernetes The Hard Way
https://github.com/kelseyhightower/kubernetes-the-
hard-way
2. Design Patterns for Container-based Distributed
Systems by Brendan Burns and David Oppenheimer,
Google
https://www.usenix.org/node/196347
3. Site Reliability Engineering: How Google Runs
Production Systems
http://shop.oreilly.com/product/0636920041528.do

102. 102
Read (2 of 2)
4. Kubernetes Cluster Federation (previously "Ubernetes")
https://github.com/kubernetes/kubernetes/blob/
release-1.5/docs/design/federated-services.md
5. From Google to the world: the Kubernetes origin story
https://cloudplatform.googleblog.com/2016/07/from-
Google-to-the-world-the-Kubernetes-origin-story.html
6. Minikube is a tool that makes it easy to run Kubernetes
locally. Minikube runs a single-node Kubernetes cluster
inside a VM on your laptop.
https://kubernetes.io/docs/getting-started-guides/
minikube/

103. 103
Observe
1. https://twitter.com/kubeconio
2. https://twitter.com/kubernetesio
3. https://twitter.com/kelseyhightower
4. https://twitter.com/kubernetesnyc
5. https://twitter.com/kubernetesber
6. https://github.com/kubernetes-incubator
7. https://twitter.com/kubernetesmsk

104. Welcome to the club

105. Questions?
@DenisIzmaylov
https://github.com/DenisIzmaylov
denis_izmaylov
Denis Izmaylov
Send“Hello”to:
izmaylov.dm@gmail.com