- 발표자: 라인 이어형 - 상세 정보: https://event.openinfradays.kr/2018/workshop2/30-track-4

1. immutable
kubernetes architecture
LINE, eohyung lee(liquidnuker@gmail.com)

2. Who am I?
이어형 (a.k.a 어형부형)
현재 LINE: cloud native service 들을 설계/개발/운영 by kubernetes
이전 kakao: private cloud 설계/개발/운영 by openstack 7+ projects
이전 kt: public cloud storage 설계/개발/운영 by openstack swift

3. 오늘의주제
좀더 단순하고
좀더 변경이쉽고
좀더 애자일하고
좀더 관리하기 편한
kubernetes(이하k8s) 아키텍쳐를고민해봅시다
✨linuxkit✨

4. 1. k8s 관리방법

5. 다양한k8s 관리방법
less exibility
more manage cost
solutions
↑ public cloud(AKS, EKS, GKE)
kubicorn, kops
kubespray
↓ kubeadm

6. exibility
내가 원하는 옵션들을 마음대로 넣고
내가 원하는 버전을 사용하고
내가 원하는 아키텍쳐를 사용할 수 있는

7. manage cost
직접 디플로이하기 위한 코드를 작성
직접 장애시 트러블 슈팅
직접 운영이슈 해결

8. exibility vs manage cost
택시vs 자가용

9. 결국선택은자신이편한것, 상황에맞는것
public cloud
AKS, EKS, GKE
kubicorn, kops
kubespray
kubeadm
private cloud
kubespray
kubeadm

10. 2. 인프라관리방법들

11. -- Chad Fowler - Trash Your Servers and Burn Your Code: Immutable Infrastructure and
Disposable Components
시스템관리자로서내가가장무서워하는것중하나는
오래동안시스템및응용프로그램을여러번업그레이드
한서버입니다.
왜? 오래된시스템은필연적으로안보이는문제를키우기
때문입니다.
“
“

12. 인프라를관리방법들
by hands(scripts)
by code(infrastructure as code)
by image(immutable infrastructure)

13. by hands(scripts)
shell scripts
cssh(cluster ssh)
makes snow ake server
모든 눈송이의 모양은 고유함
즉, 문제들의 재연성이 떨어짐

14. by code(infrastructure as code)
a.k.a programmable infrastructure
con guration management
chef, ansible, puppet, saltstack, ...
ochestration
heat, cloudformation, terraform

15. 코드기반으로관리할때장점
간단하게 추가 하기 쉬움
처음 사용하는 사람도 쉽게 관리 할 수 있음
현재 인프라에서 무엇이 변경되었는지 알기 쉬움

16. 코드기반으로관리하는솔루션들의힘든점
배포시 외부요인으로 실패 할 수 있음
롤백이 힘듬
절차적 구조기반일수록 중간에 실패시 특정 이벤트가 무시되어 질 수가 있음
선언적 구조기반일수록 복잡한 시나리오 대응이 힘듬

17. 배포시외부요인으로실패할수있음
외부 레포지토리가 깨짐, 접근 안됨, 패키지가 없어짐 등등
$ curl -v https://kubernetes-helm.storage.googleapis.com/helm-v2.6.2-linux-amd64.tar.gz
* Trying 172.217.25.208...
* TCP_NODELAY set
* Connection failed
* connect to 172.217.25.208 port 443 failed: Connection refused
* Failed to connect to kubernetes-helm.storage.googleapis.com port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to kubernetes-helm.storage.googleapis.com port 443: Connection refused
오늘의 배포가 예측 못하게 실패 할 수 있으며 stage에서 재연을 못할 수 있음

18. 롤백이힘듬
대부분 롤백은 그에 상응 하는 반대의 코드가 있어야함
파일이 생겼으면 파일이 삭제 되는 상태의 코드가 필요
파일이 업데이트 되었으면 파일이 업데이트 이전 상태가 될수 있는 코드가 필
요
모든 코드를 작성하면서 롤백 코드를 작성하는것은 무리
결국 수작업으로 반대 코드에 상응한 작업을 진행
결국 snow akeserver를 만듬

19. 절차적구조기반일수록실패시특정이벤트가무시될수있음
1. 최초 실행
A(파일 변경)
B(문제 있는 이벤트) X 실패
C(A가 변경시 프로세스 리스타트) 실패되어 실행 안됨
2. 이후 실행
A(파일이 이미 변경되어 변경이 안되고 C이벤트 트리거가 안됨)
B(문제 있는 이벤트지만 이번엔 성공)
C(A가 변경 안되서 트리거가 안됨)

20. 선언적구조기반일수록복잡한시나리오대응이힘듬

21. 계속패치된인프라와새인프라의차이

22. 여러버전의인프라상태

23. 오래된서버와새로운서버는다름

24. 오래된서버를주기적으로제거

25. -- Martin Fowler - PhoenixServer
주기적인간격으로당신의서버를불태우는것은좋은생
각입니다.
서버는불사조같이반드시주기적으로재에서부터살아
날것입니다.
“
“

26. by image(immutable infrastructure)
packer
unikernel
linuxkit

27. mutable vs immutable
From https://github.com/cetic/unikernels

28. immutable 한배포방법

29. immutable은code와data를분리하기위함
code (immutable)
application data (mutable)

30. packer
+ chef, ansible, saltstack, puppet
하지만 leaner, secure, portable한가?

31. unikernel
-- What are Unikernels - unikernel.org
-- Alfred Bratterud - #includeOS
From https://mjbright.github.io/Talks/2017-Jul-RMLL-Unikernels-WhatUsage/#5
Unikernels are specialized, single-address-space machine images
constructed by using library operating systems
“
“
“VM은 무겁지 않습니다. OS가 그렇죠"“ “

32. vm/containers/containers in vms/unikernel
From https://www.ericsson.com/research-blog/unikernels-meet-nfv/

33. container 에서unikernel 로변천과정
From http://unikernel.org/blog/2015/unikernels-meet-docker

34. 이런unikernel을어디서사용하고있는가
hyperkit, vpnkit
From https://blog.docker.com/2016/05/docker-unikernels-open-source/

35. unikernel은조금더사용성이좋아져야함
Technology Cons
Unikernels
- Not mature enough yet for production
- Requires developing applications from the grounds up
- Limited deployment possibilities
- Lack of complete IDE support
- Static resource allocation
- Lack of orchestration tools
From https://github.com/cetic/unikernels

36. 3. linuxkit

37. A toolkit for building custom minimal,
immutable Linux distributions.

38. Secure defaults without compromising usability
지정된 container 이미지만 사용하여 구성된 OS를 이용하여 보안에 우수
Everything is replaceable and customisable
모든 파트는 container로만 구성되어 원하면 변경, 교체가 아주 쉬움
Immutable infrastructure applied to building Linux distributions
code로 부터 생성된 rootfs는 immutable(read only lesystem)임
custom linux 배포판임

39. Completely stateless, but persistent storage can be attached
code와 data의 분리로 data는 추가적인 스토리지(or 디스크를 사용)
Easy tooling, with easy iteration
빌드, 배포 등이 간편
Built with containers, for running containers
컨테이너로 만들어지며 컨테이너로 돌아감

40. Designed for building and running clustered applications, including
but not limited to container orchestration such as Docker or
Kubernetes
clustered applications을 만들고 돌리기 위한 디자인
Designed from the experience of building Docker Editions, but
redesigned as a general-purpose toolkit
docker를 사용하는 경험을 그대로 가져가려고 함
Designed to be managed by external tooling, such as Infrakit or
similar tools
외부 생태계와 연계

41. linuxkit architecture
From https://www.slideshare.net/Docker/linuxkit-deep-dive

42. linuxkit yaml to architecture

43. linuxkit command
$ linuxkit --help
USAGE: linuxkit [options] COMMAND
Commands:
build Build an image from a YAML file
metadata Metadata utilities
pkg Package building
push Push a VM image to a cloud or image store
run Run a VM image on a local hypervisor or remote cloud
serve Run a local http server (for iPXE booting)
version Print version information
help Print this message
Run 'linuxkit COMMAND --help' for more information on the command
Options:
-q Quiet execution
-v Verbose execution

44. linuxkit 주요command
Commands:
build Build an image from a YAML file
pkg Package building
push Push a VM image to a cloud or image store
run Run a VM image on a local hypervisor or remote cloud

45. linuxkit build
$ linuxkit build
Please specify a configuration file
USAGE: linuxkit build [options] <file>[.yml] | -
Options:
-dir string
Directory for output files, default current directory
-disable-content-trust
Skip image trust verification specified in trust section of config (default false)
-format value
Formats to create [ aws docker dynamic-vhd gcp iso-bios iso-efi kernel+initrd
kernel+squashfs qcow2-bios qcow2-efi raw-bios raw-efi rpi3 tar tar-kernel-initrd vhd vmdk ]
-name string
Name to use for output files
-o string
File to use for a single output, or '-' for stdout
-pull
Always pull images
-size string
Size for output image, if supported and fixed size (default "1024M")

46. linuxkit build
From https://techblog.rakuten.co.jp/2017/07/26/moby-and-linuxkit/

47. linuxkit yaml
https://github.com/moby/tool/blob/master/docs/yaml.md

48. kernel
kernel:
image: linuxkit/kernel:4.14.51
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0
kernel speci es a kernel Docker image, containing a kernel and a
lesystem tarball, eg containing modules.
https://github.com/linuxkit/linuxkit/blob/master/docs/kernels.md
커스텀 커널로 패치해서 빌드하기 용의

49. init
init:
- linuxkit/init:923355a5ed75f40ee394741829d42bf4b78cce46
- linuxkit/runc:v0.4
- linuxkit/containerd:f2bc1bda1ab18146967fa1a149800aaf14bee81b
init is the base init process Docker image, which is unpacked as the
base system, containing init, containerd, runc and a few tools.

50. onboot
onboot:
- name: dhcpcd
image: linuxkit/dhcpcd:v0.4
command: ["/sbin/dhcpcd", "--nobackground", "-f", "/dhcpcd.conf", "-1"]
onboot are the system containers, executed sequentially in order.
They should terminate quickly when done.

51. service
services:
- name: getty
image: linuxkit/getty:44730fd0a7c59dbacf5b48b54ba33f551bcf7ef0
env:
- INSECURE=true
- name: redis
image: redis:4.0.5-alpine
capabilities:
- CAP_NET_BIND_SERVICE
- ...
net: host
services is the system services, which normally run for the whole
time the system is up

52. linuxkit pkg
$ ls -l
total 12
-rw-r--r-- 1 al staff 469 5 17 01:49 Dockerfile
-rw-r--r-- 1 al staff 159 4 28 10:50 build.yml
-rw-r--r-- 1 al staff 1168 4 12 09:50 dhcpcd.conf
drwxr-xr-x 3 al staff 96 4 12 09:50 usr
$ linuxkit pkg
USAGE: linuxkit pkg [subcommand] [options] [prefix]
'subcommand' is one of:
build
push
show-tag
$ linuxkit pkg build pkg/dhcpcd/

53. linuxkit pkg Docker le
$ cat Dockerfile
FROM linuxkit/alpine:1b05307ae8152e3d38f79e297b0632697a30c65c AS mirror
RUN mkdir -p /out/etc/apk && cp -r /etc/apk/* /out/etc/apk/
RUN apk add --no-cache --initdb -p /out
alpine-baselayout
busybox
dhcpcd
musl
# Remove apk residuals
RUN rm -rf /out/etc/apk /out/lib/apk /out/var/cache
FROM scratch
ENTRYPOINT []
CMD []
WORKDIR /
COPY --from=mirror /out/ /
COPY /dhcpcd.conf /usr/ /
CMD ["/sbin/dhcpcd", "--nobackground", "-f", "/dhcpcd.conf"]

54. linuxkit pkg build.yml
$ cat build.yml
image: dhcpcd
config:
binds:
- /run/resolvconf:/etc
capabilities:
- CAP_NET_ADMIN
- CAP_NET_BIND_SERVICE
- CAP_NET_RAW
- CAP_SYS_ADMIN

55. linuxkit pkg build

56. linuxkit push
$ linuxkit push --help
USAGE: linuxkit push [backend] [options] [prefix]
'backend' specifies the push backend.
Supported backends are
aws
azure
gcp
openstack
packet
vcenter

57. linuxkit run
$ linuxkit run --help
USAGE: linuxkit run [backend] [options] [prefix]
'backend' specifies the run backend.
If not specified the platform specific default will be used
Supported backends are (default platform in brackets):
aws
azure
gcp
hyperkit [macOS]
hyperv [Windows]
openstack
packet
qemu [linux]
vbox
vcenter
vmware

58. linuxkit run everywhere
From http://www.adelzaalouk.me/2017/moby-linuxkit/

59. container os와차이점은?
rancher os, core os, atomic 과의 차이점은 보다 leaner 함
그리고 linuxkit 으로 이런 container os의 custom 배포판을 생성 가능함
https://github.com/rancher/os/issues/2156

60. 4. linuxkit + k8s

61. linuxkit kubernetes kernel, init
kernel:
image: linuxkit/kernel:4.14.40
cmdline: "console=tty0 console=ttyS0"
init:
- linuxkit/init:c79d7587fcd0a195b8a3ecafe428a30e735cf2b4
- linuxkit/runc:d659de11767a419319b175700a7c6f64b8704f8c
- linuxkit/containerd:018fc633223d8a49f650da365603a5abccc6a423
- linuxkit/ca-certificates:f882e9be933fac737bf1f4d303a4bb49a12f302f

62. linuxkit kubernetes onboot
onboot:
- name: sysctl
image: linuxkit/sysctl:2a98cb7a116d4d8a71498cea0e0ad8116a9b5a3b
binds:
- /etc/sysctl.d/01-kubernetes.conf:/etc/sysctl.d/01-kubernetes.conf
readonly: false
- name: sysfs
image: linuxkit/sysfs:dc7b876f395fa44c2b93bad6b987e418497c5b34
- name: dhcpcd
image: linuxkit/dhcpcd:193a81bd4a93779c8a048d66e0cb1d201d0ae102
command: ["/sbin/dhcpcd", "--nobackground", "-f", "/dhcpcd.conf", "-1"]
- name: metadata
image: linuxkit/metadata:6962be42ec291db67ac9bb4267d8fd06fd464f48
- name: format
image: linuxkit/format:9a537dd3eaefd02dbc01c598b843fba33da8c1a5
- name: mounts
image: linuxkit/mount:a718496fa5ea2a7a9f7552eac64c7f3699fb6a86
command: ["/usr/bin/mountie", "/var/lib/"]

63. linuxkit kubernetes services
services:
- name: getty
image: linuxkit/getty:3fa8e2240c1392ba4af508d3e6be8548443b12cc
env:
- INSECURE=true
cgroupsPath: systemreserved/getty
- name: rngd
image: linuxkit/rngd:aaa9a63cbc9c04421b160b85aef4df5fa5d0f5f0
cgroupsPath: systemreserved/rngd
- name: ntpd
image: linuxkit/openntpd:413ee972bc71a66030c50bc8daf7385e5c8ea269
cgroupsPath: systemreserved/ntpd
- name: sshd
image: linuxkit/sshd:5544de2376475f6685e12bdc10bfe49f4695873a
cgroupsPath: systemreserved/sshd
- name: kubelet
image: linuxkit/kubelet:bcff9a97b3ecaf204ca7ee122f5a071a3dc97d4c
cgroupsPath: podruntime/kubelet

64. linuxkit kubernetes les
files:
- path: etc/linuxkit.yml
metadata: yaml
- path: /etc/kubernetes
symlink: "/var/lib/kubeadm"
- path: /etc/os-release
contents: |
PRETTY_NAME="LinuxKit Kubernetes Project"
- path: /usr/libexec/kubernetes/kubelet-plugins
symlink: "/var/lib/kubelet-plugins"
- path: /etc/kubeadm/
directory: true
- path: /etc/sysctl.d/01-kubernetes.conf
contents: 'net.ipv4.ip_forward = 1'
- path: /etc/cni/net.d
directory: true
- path: /opt/cni/bin
directory: true
- path: root/.ssh/authorized_keys
source: ~/.ssh/id_rsa.pub
mode: "0600"
optional: true

65. linuxkit kubernetes docker
services:
- name: docker
image: docker:18.03.0-ce-dind
capabilities:
- all
pid: host
mounts:
- type: bind
source: /sys/fs/cgroup
destination: /sys/fs/cgroup
options: ["rw","rbind","rshared","nosuid","noexec","nodev","relatime"]
binds:
- /dev:/dev
- /etc/resolv.conf:/etc/resolv.conf
- /etc/os-release:/etc/os-release
- /lib/modules:/lib/modules
- /run:/run
- /var:/var:rshared,rbind
- /var/lib/kubeadm:/etc/kubernetes
- /var/lib/cni/conf:/etc/cni/net.d:rshared,rbind
- /var/lib/cni/bin:/opt/cni/bin:rshared,rbind
- /var/lib/kubelet-plugins:/usr/libexec/kubernetes/kubelet-plugins:rshared,rbind

66. linuxkit kubernetes docker (cont.)
rootfsPropagation: shared
command: ["/usr/local/bin/docker-init", "/usr/local/bin/dockerd"]
runtime:
mkdir: ["/var/lib/kubeadm", "/var/lib/cni/conf", "/var/lib/cni/bin", "/var/lib/kubelet-plugins"
cgroupsPath: podruntime/docker
- name: kubernetes-docker-image-cache-common
image: linuxkit/kubernetes-docker-image-cache-common:2da94714...
cgroupsPath: podruntime/common-cache
files:
- path: /etc/kubelet.sh.conf
contents: ""

67. Demo
https://github.com/leoh0/kubernetes

69. 결국
k8s infra는immutable (code)
k8s에올라가는app은mutable (data)

70. 5. linuxkit + k8s 관리방법

71. node 추가
노드 bootup시 해야 할일
master : kubeadm init
HA master : kubeadm init /w CA cert
node : kubeadm join

72. node 삭제
$ kubectl drain kube-node --ignore-daemonsets --force --delete-local-data
(linuxkit kube-node)$ poweroff
$ kubectl delete node kube-node

73. check graceful termination
https://cloudplatform.googleblog.com/2018/05/Kubernetes-best-
practices-terminating-with-grace.html
1. Pod is set to the “Terminating” State and removed from the
endpoints list of all Services
2. preStop Hook is executed
3. SIGTERM signal is sent to the pod
(만약 안될시 https://github.com/krallin/tini )
4. Kubernetes waits for a grace period
5. SIGKILL signal is sent to pod, and the pod is removed

74. upgrade k8s
https://kubernetes.io/docs/tasks/administer-
cluster/kubeadm/kubeadm-upgrade-ha/
upgrade control plane(api,controller,schduler, ...)
upgrade static pod
or add new master
upgrade node agent(kubelet)
add new node

75. 좀더관리를단순화시켜본다면

76. self hosting k8s
master node 의 static pod들을 daemon-set, deployment화 하여 node
관리와 cluster lifecycle 관리를 원활하게 함
$ kubectl -n kube-system get deployments
NAME DESIRED CURRENT
kube-controller-manager 2 2
kube-dns 1 1
kube-scheduler 2 2
$ kubectl -n kube-system get daemonsets
NAME DESIRED CURRENT NODE SELECTOR
kube-apiserver 1 1 node-role.kubernetes.io/master=
$ kubectl -n kube-system get secrets
NAME TYPE
kube-apiserver Opaque
kube-controller-manager Opaque

77. self hosting in kubeadm
# kubeadm alpha phase selfhosting --help
This command is not meant to be run on its own. See list of available subcommands.
Usage:
kubeadm alpha phase selfhosting [command]
Aliases:
selfhosting, selfhosted, self-hosting
Available Commands:
convert-from-staticpods Converts a static Pod-hosted control plane into a self-hosted one
Use "kubeadm alpha phase selfhosting [command] --help" for more information about a command.

78. bootstrapping self hosting
From Self-Hosted Kubernetes How and Why

79. bootstrapping self hosting (cont.)
From Self-Hosted Kubernetes How and Why

80. bootstrapping self hosting (cont.)
From Self-Hosted Kubernetes How and Why

81. disaster recovery self hosting
control plane의 상태를 파악해서 문제시 static pods을 띄워주는 daemon이
있음
From Self-Hosted Kubernetes How and Why

82. self hosting이적용되었을시
node 추가
master : kubectl label node n1 master=true
node : kubeadm join
upgrade
upgrade control plane:
kubectl set image ds/kube-apiserver kube-
apiserver=gcr.io/google_containers/kube-apiserver-amd64:v1.9.6
update node agent: node 추가

83. cluster life cycle management like escalator

84. cluster life cycle management

85. 그렇다면kubernetes는immutable 한가?
Let's also burn old kubernetes.
오래된 k8s는 지우고 새로운 k8s로 데이터 sync(not migrated)
federation
https://kubernetes.io/docs/concepts/cluster-
administration/federation/

86. Thank you