As part of the long term goal to contribute to the Kubernetes ecosystem of components and tools, at Databricks we've implemented a Python tool called kubecfg to allow us to succinctly describe deployment configurations in a highly modular way. In its current version, kubecfg will compile Jsonnet files using the Python _jsonnet library from Google, accepting command-line arguments to override specific fields in a generated dictionary or to pass arguments to a Jsonnet script that accepts top-level arguments.

1. A Tale of Three Tools:
Kubernetes, Jsonnet, and Bazel
Miles Yucht
SF Kubernetes Meetup
July 26, 2017 1

2. Who am I?
Miles
• CS @ Princeton
• SWE on Growth team @ Databricks
since 2015
• "Lead Developer" of kubecfg
2

3. Agenda
• Kubernetes @ Databricks
• Why Jsonnet
• Combining Bazel and Kubernetes
• Q & A
3

4. Kubernetes
4

5. Kubernetes at Databricks
● Multiple Kubernetes clusters, hundreds of deployments
● 6+ terabytes of memory, hundreds of cores
● Many customizations for security
● Everything new goes onto Kubernetes
● All managed by a 7-person team
5

6. Kubernetes at Databricks
We're hiring! Come talk to
Priya or me after the talks!
6

7. Combinatorial Configurations
6 clusters, 3 environments, 4 modes of deployment
● Lots of parameters
● Extremely verbose
○ >75k lines of checked-in YAML templates
● YAML is not modular
● Hard to manage, manipulate, and correctly refactor templates
7

8. Kubernetes development lifecycle
● Simple to test & develop in Kubernetes
● Keep dev, test, staging, prod environments similar
● Critical for testing kubernetes/cloud features
○ E.g. replication, PROXY protocol
8

9. Many-step Deployment
9
Build Docker Image
Tag Docker Image
Push Docker ImageEdit Template
Apply Template

10. kubecfg!
● Python wrapper around kubectl
○ Still have all your favorite kubectl commands
● Preprocesses inputs and passes the result to kubectl
● Augments kubectl
○ Easy to add new commands to kubecfg
● Automates several common workflows
○ Simpler for developers
○ Fewer chances to make mistakes
10

11. Jsonnet
11

12. Jsonnet
● DSL for defining JSON data
● Break up monolithic configurations into composable parts
● >75k lines of YAML → ~20k lines of Jsonnet
● To learn about how we use Jsonnet:
○ https://databricks.com/blog/2017/06/26/declarative-infrastructure-js
onnet-templating-language.html
12

13. Jsonnet → JSON
When running kubecfg apply -f webapp.jsonnet:
● jsonnet webapp.jsonnet > /tmp/aaa.json
● kubectl apply -f /tmp/aaa.json
13

14. Original kubecfg architecture
14
Jsonnet Template Kubernetes spec Kubectl

15. Integrating build and deploy
Kubernetes template
⇕
Tagged image
15

16. Integrating build and deploy
Kubernetes template
⇕
How to produce the artifact that it depends on
16

17. Integrating build and deploy
1. What do you need to do to prepare the artifact?
→ Ask your build tool.
2. How do you refer to the artifact?
→ Use a tagged docker image.
17

18. Bazel
18

19. Bazel
● Open-source, language-agnostic build tool
● Very customizable/configurable
● Ported our build from SBT to Bazel mid-2016
● To learn about how we use Bazel:
○ Blog post coming soon!
19

20. Kubectl Templates
{
image: "registry.databricks.com/webapp:latest",
...
}
20

21. Kubectl Templates
{
image: "registry.databricks.com/webapp:latest",
...
}
Kubernetes template
⇕
Tagged image
21

22. Kubecfg Templates
{
target: "//docker/webapp",
...
}
22

23. Kubecfg Templates
{
target: "//docker/webapp",
...
}
Kubernetes template
⇕
Bazel target
23

24. Integrating build and deploy
Build:
● bazel build //docker/webapp:webapp
Tag & Push (new):
● bazel run //docker/webapp:webapp_push -t <tag>
Deploy (new):
● $(cat //docker/webapp:webapp_repository):<tag>
24

25. Tagging images
For clean checkouts:
● <GIT_SHA>-<COMMIT_TIMESTAMP>
For dirty checkouts:
● <GIT_SHA>-<COMMIT_TIMESTAMP>-DIRTY-<DIFF_SHA>
25

26. Current kubecfg architecture
26
Jsonnet Template
“Kubernetes spec”
with Bazel targets
Kubectl
Kubernetes spec
with images
Build docker
image
Tag and
push image
git ref-log

27. Conclusion
● kubecfg makes deployment a two-step operation
1. git checkout my-branch
2. kubecfg apply -f the-template.jsonnet
● Speeds up development cycle
● Helps with tracking down errors
27

28. Future of kubecfg
● We plan on open sourcing this library
● Development in Bazel makes this easier
○ https://github.com/bazelbuild/rules_docker
28

29. Thank you
We're hiring! Talk to me or Priya if you're
interested in joining our team.
29