Talk held on R-ETAIL:CODE in London on 2018-03-15. - The history of how DevOps evolved at Zalando: from on-premise data centers to autonomous teams, microservices and cluster management in the cloud - How the developer experience looks like for the application lifecycle from idea to production and what our vision for the future is - Challenges and learnings from our past experiences: why architecture principles and constraints are important to lead 200+ engineering teams

1. R-ETAIL:CODE
2018-03-15
HENNING JACOBS
@try_except_
Developer Journey
at Zalando

2. 2
ZALANDO IN NUMBERS
> 4.4billion EUR
2017
> 200
million
visits
per
month
> 14,000
employees in
Europe
> 70%
of visits via
mobile devices
> 23
million
active customers
> 250,000
product choices
~ 2,000
brands
15
countries

3. 3
OUR FOOTPRINT AROUND EUROPE
as of November 2017
1
8
10
11
12
13
BERLIN HEADQUARTERS AND OUTLET
BRIESELANG FULFILLMENT CENTER
ERFURT FULFILLMENT CENTER AND TECH OFFICE
MÖNCHENGLADBACH FULFILLMENT CENTER AND TECH OFFICE
LAHR FULFILLMENT CENTER
DORTMUND TECH HUB
FRANKFURT OUTLET
DUBLIN TECH HUB
HELSINKI TECH HUB
MILAN (STRADELLA) FULFILLMENT CENTER
KÖLN OUTLET
PARIS (MOISSY-CRAMAYEL) FULFILLMENT CENTER
SZCZECIN (GRYFINO) FULFILLMENT CENTER
HAMBURG ADTECH LAB
STOCKHOLM (BRUNNA) FULFILLMENT CENTER
10
9
7
6
5
3
2
1
11
12
13
4
14
15
15
14
9
8
7
6
5
4
3
2
1

4. 4
OUR FOOTPRINT AROUND EUROPE
TECH
as of November 2017
1
8
10
11
12
13
BERLIN HEADQUARTERS AND OUTLET
BRIESELANG FULFILLMENT CENTER
ERFURT FULFILLMENT CENTER AND TECH OFFICE
MÖNCHENGLADBACH FULFILLMENT CENTER AND TECH OFFICE
LAHR FULFILLMENT CENTER
DORTMUND TECH HUB
FRANKFURT OUTLET
DUBLIN TECH HUB
HELSINKI TECH HUB
MILAN (STRADELLA) FULFILLMENT CENTER
KÖLN OUTLET
PARIS (MOISSY-CRAMAYEL) FULFILLMENT CENTER
SZCZECIN (GRYFINO) FULFILLMENT CENTER
HAMBURG ADTECH LAB
STOCKHOLM (BRUNNA) FULFILLMENT CENTER (start winter 2017)
10
9
7
6
5
3
2
1
11
12
13
4
14
15
15
14
9
8
7
6
5
4
3
2
1

5. 5
WE ARE CONSTANTLY INNOVATING TECHNOLOGY
HOME-BREWED,
CUTTING-EDGE
& SCALABLE
technology solutions
~ 1,900
employees from
tech locations
(HQs in Berlin)7
77
nations
help our brand to
WIN ONLINE

6. 6
ZALANDO TECH’S
INFRASTRUCTURE

7. 7
ZALANDO TECH PLATFORM
THE COMPLETE HISTORY (ABRIDGED)
ZOMCATPHP STUPS KUBERNETES
2010 2015 2016
Data center
Java/WAR
AWS
Docker
Cloud Formation
Low level (AWS API)
AWS
Docker
Cloud Formation
Kubernetes manifest
Higher abstraction level
Data center
PHP files
2008

8. 8
SCALE?
66
Clusters
329
Accounts

9. 9
ISOLATED AWS ACCOUNTS
Internet
*.xpto.example.org
Product XYZ
xpto
Account
Load Balancer
xpti
Account
Load Balancer
*.xpti.example.org

10. 10
DEVELOPER PRODUCTIVITY
Cloud Native Application Runtime
• AWS
• Kubernetes
• Continuous Delivery
Developer Tooling
• Git
• Artifact storage
• Testing Tools & Infrastructure

12. 12
KPIs
• Adoption
• Deployments per developer
• Time from commit to production
• Developer satisfaction
• Infrastructure cost efficiency
⇒ Measurable OKRs

13. 13
DEVELOPER JOURNEY
Consistent story
that models
all aspects of SW dev

14. 14
Developer
Journey

16. 16
PLAN & SETUP

17. 17
Plan
Stories
Rules of Play
Tech Radar

19. 19
Setup
Application
Bootstrapping

22. 22
BUILD & TEST

23. 23
CDP
BUILD PIPELINE
CDPGHE
trigger buildpush code
push

25. 25
DEPLOY

26. 26
Deploy
Kubernetes

27. 27
DEPLOYMENT CONFIGURATION
.
├── deploy/apply
│ ├── deployment.yaml # K8s Deployment
│ ├── credentials.yaml # K8s CRD
│ ├── ingress.yaml # K8s Ingress
│ └── service.yaml # K8s Service
└── delivery.yaml # pipeline config

28. 28
INGRESS.YAML
kind: Ingress
metadata:
name: "..."
spec:
rules:
# DNS name your application should be exposed on
- host: "myapp.foo.example.org"
http:
paths:
- backend:
serviceName: "myapp"
servicePort: 80

29. 29
CDP
DEPLOYMENT PIPELINE
CDPGHE
xpto
Account
deploytrigger deploypush code
pull

30. 30
CONTINUOUS DELIVERY PLATFORM

31. 31
CDP: APPLY

32. 32
CDP: OPTIONAL APPROVAL

33. 33
INTEGRATIONS

34. 34
CLOUD FORMATION VIA CI/CD
.
├── deploy/apply
│ ├── deployment.yaml # K8s Deployment
│ ├── cf-iam-role.yaml # AWS IAM Role
│ ├── cf-rds.yaml # AWS RDS Database
│ ├── kube-ingress.yaml # K8s Ingress
│ ├── kube-secret.yaml # K8s Secret
│ └── kube-service.yaml # K8s Service
└── delivery.yaml # CI/CD config

35. 35
ASSIGNING AWS IAM ROLE TO POD
kind: Deployment
spec:
template:
metadata:
annotations:
# annotation for kube2iam
iam.amazonaws.com/role: "app-myapp-role"
spec:
containers:
- name: ...
...
https://github.com/jtblin/kube2iam
⇒ AWS SDKs just work as expected

36. 36
POSTGRES OPERATOR
Application to manage PostgreSQL clusters
Observes “postgres” manifests (CRDs)
Spawns and modifies new clusters
Syncs and provisions roles
Handles volume resize, incl. Resize2fs
Also responsible for updating Docker images

37. 37
SUMMARY
• Application Bootstrapping
• Git (and GHE) as source of truth and UI
• 4-eyes principle for master/production
• Extensible Kubernetes API as primary interface
• OAuth/IAM credentials
• PostgreSQL
• CloudFormation for proprietary AWS services

38. 38
DEVELOPER
SATISFACTION

40. 40
RESULTS
Response rate: 16%
NPS, average, median, index?

41. 41
DOCUMENTATION
"Documentation is hard to find"
"Documentation is not comprehensive enough"
"Remove unnecessary complexity and obstacles."
"Get the documentation up to date and prepare
use cases"
"More and more clear documentation"
"More detailed docs, example repos with more
complicated deployments."

42. 42
DOCUMENTATION
• Restructure following
https://www.divio.com/en/blog/documentation/
• Concepts
• How Tos
• Tutorials
• Reference
• Global Search
• Weekly Health Check: Support → Documentation

44. 44
LEARNINGS
FROM THE PAST

45. 45
LEARNINGS FROM THE PAST
• Platform as a service vs governance
• Platform teams think they are "special",
but lack basic customer insights
• Developers love autonomy, hard to "go back" once granted
• Developers are demanding

46. 46
NOTE TO SELF
• Establish a product mindset
• Listen to your customers
• Exploit unfair advantages
• Invest into documentation & support

47. 47
FUTURE

48. 48
FUTURE
• Kubernetes adoption
• Multi cloud strategy
• More streamlined end-to-end experience (monitoring, ..)
• Mindshift towards full hands-off operations
• Data science use cases

49. 49
OPEN SOURCE
Running Kubernetes in Production on AWS
http://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-production.html
Kube AWS Ingress Controller
https://github.com/zalando-incubator/kube-ingress-aws-controller
External DNS
https://github.com/kubernetes-incubator/external-dns
PostgreSQL Operator
https://github.com/zalando-incubator/postgres-operator
Zalando Cluster Configuration
https://github.com/zalando-incubator/kubernetes-on-aws

50. QUESTIONS?
HENNING JACOBS
HEAD OF
DEVELOPER PRODUCTIVITY
henning@zalando.de
@try_except_
Illustrations by @01k