How Docker is used in the open source STUPS cloud infrastructure @ Zalando Tech.

1. Docker
Powering Radical Agility
Docker Berlin Meetup 2015-06-23
henning.jacobs@zalando.de @try_except_

2. DOCKER-BASED DEPLOYMENT
STUPS CLOUD PLATFORM
HISTORY
RADICAL AGILITY & ARCHITECTURE PRINCIPLES
AGENDA

3. 15 countries
3 fulfillment centers
15+ million active customers
2.2+ billion € revenue 2014
130+ million visits per month
8.000+ employees
ONE OF EUROPE’S LARGEST ONLINE FASHION RETAILERS
Visit us: tech.zalando.com

4. A BRIEF
HISTORY OF
ZALANDO
TECHNOLOGY

5. A BRIEF HISTORY OF ZALANDO TECH

6. ZALANDO PLATFORM

7. ~70% of all applications
WAR deployment
Single deployment tool
On-premise data center
MAIN PRODUCTION STACK SINCE 2010

8. Platform
THE CHALLENGE
Platform team
request servers
deploy

9. Platform
THE CHALLENGE
80+ delivery teams
Platform team
deploy
request servers
request storage

10. RADICAL
AGILITY

11. GOAL
DELIVER AMAZING
PRODUCTS
EFFICIENTLY AT
SCALE, AND
FEELING GREAT
ABOUT IT.

12. 3 PRINCIPLES

13. PURPOSE

14. AUTONOMY

15. MASTERY

16. ARCHITECTURE
AN
ARCHITECTURE
FOR
INNOVATION

17. API FIRST

18. REST

19. SAAS

20. MICRO
SERVICES

21. CLOUD

22. STUPS
STUPS To Unleash Penguin Swarms

23. AWS
STUPS
DOCKER
DEPLOY
SSH
ACCESS
AUDIT
REPORTS
FULL AWS
ACCESS
A PLATFORM ON TOP OF AMAZON WEB SERVICES

24. AUTONOMY AND COMPLIANCE
STUPS offers
maximum freedom for developers
while enabling
near-real-time audit compliance
for every single application.

25. One AWS account per Team
Deployment with Docker
Managed SSH Access
REST/OAuth 2.0 mandatory
Supports Traceability of Changes
STUPS IN A NUTSHELL

26. Public Internet
*.foo.example.org *.bar.example.org
Team “Foo” Team “Bar”ELB ELB
EC2
Instance
EC2
InstanceEC2
InstanceEC2
Instance
EC2
InstanceEC2
Instance
Data Center LB
EC2
InstanceEC2
InstanceLegacy
Instances
ISOLATED AWS ACCOUNTS

27. DEPLOYMENT

28. IMMUTABLE STACKS

29. AWS
DEPLOYMENT WITH SENZA
Senza CLI
Pier One
docker pull
docker push
Taupage

30. FROM zalando/openjdk:8u40-b09-4
EXPOSE 8080
COPY target/hello-world.jar /
COPY target/scm-source.json /
CMD java $(java-dynamic-memory-opts) ↲
-jar /hello-world.jar
DOCKERFILE

31. $ docker build -t ↲
pierone.example.org/myteam/hello-world:0.2 .
$ pierone login
Getting OAuth2 token "pierone".. OK
Storing Docker client configuration in ~/.dockercfg.. OK
$ docker push pierone.example.org/myteam/hello-world:0.2
DOCKER BUILD & PUSH

32. $ pierone tags myteam hello-world
Team │Artifact │Tag │Created│By |
myteam hello-world 0.1-andre-test 13d ago ahartmann
myteam hello-world 0.1 3d ago ahartmann
myteam hello-world 0.2 3m ago hjacobs
$ pierone scm myteam hello-world 0.2
Tag│Author │URL │Revision │Status│Created│By
|
0.2 hjacobs git:git@github.. 442b7502 10m ago hjacobs
VERIFY IMAGE UPLOAD

33. PIER ONE DOCKER REGISTRY
✓ S3 backend to store images
✓ OAuth2 integration
✓ Team repositories
✓ Immutable tags & scm-source.json
✓ JVM-based (Clojure)
✓ Command line interface (Python)

34. SENZA: DEFINITION YAML
SenzaInfo:
StackName: hello-world
Parameters:
- ImageVersion:
Description: "Docker image version of Hello World."
SenzaComponents:
- Configuration:
Type: Senza::StupsAutoConfiguration # auto-detect network setup
- AppServer: # will create a launch configuration and ASG with scaling triggers
Type: Senza::TaupageAutoScalingGroup
InstanceType: t2.micro
SecurityGroups: [app-hello-world]
ElasticLoadBalancer: AppLoadBalancer
TaupageConfig:
runtime: Docker
source: "stups/hello-world:{{Arguments.ImageVersion}}"
ports:
8080: 8080

35. SENZA: STACK DEPLOYMENT
$ senza create hello-world.yaml 1 0.2
Generating Cloud Formation template.. OK
Creating Cloud Formation stack hello-world-1.. OK
$ senza events hello-world.yaml 1
Stack Name│Ver.│Resource Type │Resource ID │Status │Status Reason │Event
Time
hello-world 1 CloudFormation::Stack hello-world-1 CREATE_IN_PROGRESS User Initiated 10m ago
...
hello-world 1 CloudFormation::Stack hello-world-1 CREATE_COMPLETE 6m ago

36. docker run -d --log-driver=syslog ↲
--restart=on-failure:10 ↲
-e DB_SUBNAME=.. ↲
-v /meta:/meta:ro ↲
-e CREDENTIALS_DIR=/meta/credentials ↲
-p 8080:8080 -p 7979:7979 ↲
-u 999 ↲
pierone.example.org/stups/pierone:0.5
TAUPAGE: DOCKER COMMAND LINE

37. SENZA: MANAGE STACKS

38. LOGGING

39. docker run .. --log-driver=syslog ..
/etc/rsyslog.d/24-application.conf
:syslogtag, startswith, "docker" ↲
/var/log/application.log
/etc/logrotate.d/..
Don’t forget log rotation..
TAUPAGE: DOCKER SYSLOG

40. APPLICATION LOGS: TAUPAGE SUPPORTS LOGENTRIES AND SCALYR

41. SSH ACCESS

42. SSH ACCESS: TIME-LIMITED ACCESS TO ANY TEAM SERVER

43. MONITORING

44. TODO: Screenshot
ZMON

45. ZMON APPLIANCE
*.foo.example.org *.bar.example.org
Team “Foo” Team “Bar”
EC2
Instance
EC2
InstanceEC2
Instance
EC2
Instance
ZMON
Appliance
ZMON
Appliance
KairosDB
EC2
Instance
EC2
Instance
ZMON
Controller
ELB ELB

46. HYSTRIX TURBINE

47. FULLSTOP: REPORT VIOLATIONS

48. DOCKER?

49. ● Ubuntu & OpenJDK base image
● Log to STDOUT
● Config via environ. vars (+ KMS decryption)
● Non-root execution
● Persistence via EBS mounts
● Immutable stacks, no orchestration
● DNS endpoints, etcd e.g. for Hystrix streams
RECAP: DOCKER IN STUPS

50. STUPS Frontpage
http://stups.io
STUPS Documentation
http://docs.stups.io
GitHub Repositories
https://github.com/zalando-stups
Trying out Senza and Taupage
http://docs.stups.io/en/latest/user-guide/standalone-deployment.html
LINKS

51. QUESTIONS?
http://stups.io
@try_except_

52. BACKUP

53. STUPS COMPONENTS

54. ● ELB for
inbound traffic
● NAT instances
for outbound
● HTTPS Only
● Internal subnets
for app instances
DMZ DMZ DMZ
internalinternal
eu-west-1a eu-west-1b eu-west-1c
ELB
EC2
internal
EC2
NAT
STUPS: AWS ACCOUNT VPC SETUP

55. Pier One Docker Reg.
build
approve
EC2 Instances
Docker
Container
Application “myapp”
issue_management: Jira
Application Version “1.0”
artifact: docker/myart:1.0
Taupage AMI
Ticket System
Kio Application Registry
Ticket System
SCM
Image “docker/myart:1.0”
commit: afb123Issue “ABC-123”
spec: [...]
Commit “afb123”
msg: ABC-123..
✓ specs approved
✓ artifact tested
✓ artifact approved
STUPS: TRACEABILITY