Creating containers for an application is easy (even if it’s a goold old distributed application like Apache Hadoop), just a few steps of packaging. The hard part isn't packaging: it's deploying How can we run the containers together? How to configure them? How do the services in the containers find and talk to each other? How do you deploy and manage clusters with hundred of nodes? Modern cloud native tools like Kubernetes or Consul/Nomad could help a lot but they could be used in different way. It this presentation I will demonstrate multiple solutions to manage containerized clusters with different cloud-native tools including kubernetes, and docker-swarm/compose. No matter which tools you use, the same questions of service discovery and configuration management arise. This talk will show the key elements needed to make that containerized cluster work. Tools: kubernetes, docker-swam, docker-compose, consul, consul-template, nomad together with: Hadoop, Yarn, Spark, Kafka, Zookeeper, Storm…. References: https://github.com/flokkr Speaker Marton Elek, Lead Software Engineer, Hortonworks

1. k8s : h4pk8s : h4p
From docker to kubernetes:
Running Apache Hadoop in a cloud native way
Marton Elek

5. RAFT library for Java

6. https://flokkr.github.io
RAFT library for Java

21. FROM frolvlad/alpine-oraclejdk8
ADD hadoop-3.2.0.tar.gz /opt
WORKDIR /opt/hadoop
DockerfileDockerfile

23. <configuration>
<property>
<name>dfs.namenode.rpc-address</name>
<value>namenode:9000</value>
</property>
<property>
<name>dfs.datanode.plugins</name>
<value>org.apache.hadoop.ozone.HddsDatanodeService</value>
</property>
<property>
<name>rpc.metrics.percentiles.intervals</name>
<value>60,300</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/data/namenode</value>
</property>
<property>
<name>rpc.metrics.quantile.enable</name>
<value>true</value>
</property>
</configuration>

24. version: "3"
services:
service1:
image: apache/imagename
hostname: namenode
ports:
- 9870:9870
environment:
CONFIGURATION1: value
DFS_DIR: /dfs
THREAD_NUMBER: 1

25. How to handle configuration?How to handle configuration?
Create a simple launcher script to
Create config file from environment variables
Start the application

26. version: "3"
services:
namenode:
image: flokkr/hadoop
hostname: namenode
command: ["hdfs","namenode"]
ports:
- 9870:9870
environment:
ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
CORE-SITE.XML_fs.defaultFS: "hdfs://namenode:9000"
HDFS-SITE.XML_dfs.namenode.rpc-address: "namenode:9000"
HDFS-SITE.XML_dfs.replication: "1"
datanode:
image: flokkr/hadoop
command: ["hdfs", "datanode"]
environment:
CORE-SITE.XML_fs.defaultFS: "hdfs://namenode:9000"
HDFS-SITE.XML_dfs.namenode.rpc-address: "namenode:9000"
HDFS-SITE.XML_dfs.replication: "1"
LOG4J.PROPERTIES_log4j.rootLogger: "INFO, stdout"
LOG4J.PROPERTIES_log4j.appender.stdout: "org.apache.log4j.ConsoleAppender"
LOG4J.PROPERTIES_log4j.appender.stdout.layout: "org.apache.log4j.PatternLayout"
LOG4J.PROPERTIES_log4j.appender.stdout.layout.ConversionPattern: "%d{yyyy-MM-dd HH:mm:ss} %-5p

27. ENV (script)
n/a
n/a

32. Container is the unit of packaging.

33. Launcher script:
the source of the power in containers

34. FROM frolvlad/alpine-oraclejdk8
ADD hadoop-3.2.0.tar.gz /opt
WORKDIR /opt/hadoop
CMD ["/opt/launcher.sh"]
DockerfileDockerfile

35. FROM frolvlad/alpine-oraclejdk8
ADD hadoop-3.2.0.tar.gz /opt
WORKDIR /opt/hadoop
CMD ["/opt/launcher.sh"]
DockerfileDockerfile
launcher.sh hdfs namenode
Final command executed by docker:

36. #!/usr/bin/env bash
if [ -n "$SLEEP_SECONDS" ]; then
echo "Sleeping for $SLEEP_SECONDS seconds"
sleep $SLEEP_SECONDS
fi
if [ -n "$ENSURE_NAMENODE_DIR" ]; then
if [ ! -d "$ENSURE_NAMENODE_DIR" ]; then
/opt/hadoop/bin/hdfs namenode -format -force $CLUSTERID_OPTS
fi
fi
"$@"

37. Launcher scriptLauncher script

38. Launcher scriptLauncher script
Create config files from ENV

39. Launcher scriptLauncher script
Create config files from ENV
Wait for the dependency (TCP check)

40. Launcher scriptLauncher script
Create config files from ENV
Wait for the dependency (TCP check)
Download additional optional component

41. Launcher scriptLauncher script
Create config files from ENV
Wait for the dependency (TCP check)
Download additional optional component
Prepare HDFS (format namenode, ...)

42. Launcher scriptLauncher script
Create config files from ENV
Wait for the dependency (TCP check)
Download additional optional component
Prepare HDFS (format namenode, ...)
Retrieve kerberos/SSL secrets

43. Launcher scriptLauncher script
Create config files from ENV
Wait for the dependency (TCP check)
Download additional optional component
Prepare HDFS (format namenode, ...)
Retrieve kerberos/SSL secrets
Enable prometheus monitoring (Java agent)

44. Launcher scriptLauncher script
Create config files from ENV
Wait for the dependency (TCP check)
Download additional optional component
Prepare HDFS (format namenode, ...)
Retrieve kerberos/SSL secrets
Enable prometheus monitoring (Java agent)
Show network traffic (Instrumentation with Java agent)

49. Hashicorp stackHashicorp stack
"do it yourself""do it yourself"

61. Docker network

62. Host network

63. docker host
host dns
dns
yes
host

64. <configuration>
<property>
<name>dfs.namenode.rpc-address</name>
<value>namenode:9000</value>
</property>
<property>
<name>dfs.datanode.plugins</name>
<value>org.apache.hadoop.ozone.HddsDatanodeService</value>
</property>
<property>
<name>rpc.metrics.percentiles.intervals</name>
<value>60,300</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/data/namenode</value>
</property>
<property>
<name>rpc.metrics.quantile.enable</name>
<value>true</value>
</property>
</configuration>

74. DIYDIY

76. Consul
Yes (script)
Restart
host netw
Nomad
.nomad
redeploy
no
yes
host netw
yes
consul
yes
host

78. KubernetesKubernetes
"out of the box""out of the box"

84. +Network!
+Storage!!
(volume, secrets,
configs)

85. DaemonSet

86. ReplicaSet

90. Service Pod

91. StatefulSet

92. DaemonSet

96. Benefits ofBenefits of
Hadoop + k8s?Hadoop + k8s?

97. Benefits ofBenefits of
Hadoop + k8s?Hadoop + k8s?
Ecosystem
Flexibility

98. Example:Example:
Monitor Hadoop with PrometheusMonitor Hadoop with Prometheus

102. Pod

103. Sidecar pattern
Pod

104. Sidecar for monitoring over jmx

105. apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: ozone-hdfs-namenode
spec:
serviceName: ozone2-hdfs-namenode
replicas: 1
template:
metadata:
labels:
app: ozone
spec:
containers:
- name: hdfs-namenode
image: flokkr/ozone:2.1.0
args: ["hdfs","namenode"]

106. apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: ozone-hdfs-namenode
spec:
serviceName: ozone2-hdfs-namenode
replicas: 1
template:
metadata:
labels:
app: ozone
spec:
containers:
- name: hdfs-namenode
image: flokkr/ozone:2.1.0
args: ["hdfs","namenode"]
Flexibility

107. apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: ozone-hdfs-namenode
spec:
serviceName: ozone2-hdfs-namenode
replicas: 1
template:
metadata:
labels:
app: ozone
spec:
shareProcessNamespace: true
containers:
- name: hdfs-namenode
image: flokkr/ozone:2.1.0
args: ["hdfs","namenode"]
- name: jmxpromo
image: flokkr/jmxpromo-sidecar

108. apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: ozone-hdfs-namenode
spec:
serviceName: ozone2-hdfs-namenode
replicas: 1
template:
metadata:
labels:
app: ozone
spec:
shareProcessNamespace: true
containers:
- name: hdfs-namenode
image: flokkr/ozone:2.1.0
args: ["hdfs","namenode"]
- name: jmxpromo
image: flokkr/jmxpromo-sidecar
Flexibility

111. apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: ozone-hdfs-namenode
spec:
serviceName: ozone2-hdfs-namenode
replicas: 1
template:
metadata:
labels:
app: ozone
spec:
containers:
- name: hdfs-namenode
image: flokkr/ozone:2.1.0
args: ["hdfs","namenode"]

112. apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: ozone-hdfs-namenode
spec:
serviceName: ozone2-hdfs-namenode
replicas: 1
template:
metadata:
labels:
app: ozone
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "28942"
spec:
containers:
- name: hdfs-namenode
image: flokkr/ozone:2.1.0
args: ["hdfs","namenode"]

114.
Ecosystem

115. configmap
helm
n/a
CNI
k8s
helm, yaml
yes
namespaces
yes
CNI
statefulset
DNS
no
service/ingress

116. Is HadoopIs Hadoop
cloud native?cloud native?

117. Is HadoopIs Hadoop
cloud native?cloud native?
Yes, with small modifications

118. Is HadoopIs Hadoop
cloud native?cloud native?
Yes, with small modifications
DNS/IP handling. Shoud work
without DNS
with changing DNS

119. Is HadoopIs Hadoop
cloud native?cloud native?
Yes, with small modifications
DNS/IP handling. Shoud work
without DNS
with changing DNS
More flexible configuration loading

120. Is HadoopIs Hadoop
cloud native?cloud native?
Yes, with small modifications
DNS/IP handling. Shoud work
without DNS
with changing DNS
More flexible configuration loading
Reverse proxy friendly UI

121. SummarySummary

122. SummarySummary
Don't buy without checking
the label

123. SummarySummary
Don't buy without checking
the label
Containerization can help a
lot to manage our Bigdata
clusters

124. SummarySummary
Don't buy without checking
the label
Hadoop is first class citizen
of cloud-native/containerized
environments*
Containerization can help a
lot to manage our Bigdata
clusters

125. Q&AQ&A
Márton Elek @anzix
https://flokkr.github.io (bigdata + containers project)
https://github.com/flokkr (source)
elek@apache.org

126. Image creditsImage credits
Yan Pritzker (CC)
Carrie Cizauskas (CC)