La startup Yuzu, accompagnée de la société Vixns, a opté pour le choix d'une infrastructure Docker tournant sur Mesos/Marathon/Consul. De l'environnement de développement jusqu'au monitoring au quotidien, nous vous ferons part de nos erreurs, réussites, workflows et outils utilisés.

1. Comment nous avons
conteneurisé une startup ?
#devopsday

2. Jonathan Martin
@john_iw2
Stéphane Cottin
@vixns

3. #architecture #hosting #opsdev #libre #r&d
#techtrends #IoT #artprojects

4. #startup #newyork #marseille #r&d #frenchtech
#ecommerce #bpi #shake #network #bigdata
#dockerlover #opensourcelover #hashtaglover

5. The first global cross-selling
network between eMerchants
“ “

6. Yuzu By Example

7. At the beginning

8. months to do a MVP
4

9. No time to reinvent the wheel

10. Symfony - Java - MongoDB

11. PRODUCTION
Web BigData
- 2 physical servers
- Multiple VMs
- Chef & Chef server
- 2 physical servers
- Stack Cloudera (Hadoop/Spark/...)

12. Six months later...

13. A lot of technos joined
Yuzu during the MVP
Welcome to nodejs, scala, elasticsearch, hbase, redis,
kafka, couchbase,...

14. They lived happily
ever after

15. It doesn’t work in production…
… that’s works fine in my local env
“ “

16. This Lib isn’t in the right version...
It’s slow… why??
Where are my logs...
“ “

17. Time to look around...

18. Be kind, rewind
Spring cleaning
Microservices / 12 factors
Resources isolation
Improved security
Continuous deployment
Start-up compliant workflow
Orchestration & supervision

19. AND
WINNERS ARE ...THE

20. Docker + Mesos + Marathon + Consul

21. {
"container": {
"type": "DOCKER",
"docker": {
"image": "redis:2.6.17",
"network": "BRIDGE",
"portMappings": [
{
"containerPort": 6379,
"hostPort": 0,
"protocol": "tcp"
}
]
}
},
"id": "redis",
"instances": 1,
"cpus": 0.5,
"mem": 1024,
"healthChecks": [
{
"protocol": "TCP",
"portIndex": 0
}
]
}
MARATHON
wants
JSON

23. Dynamic Configuration
Docker
Registrator
Consul
Consul-template

24. Logs, metrics, alerting...

25. Collect everything collectd / fluentd / rsyslog
Convert to async messages kafka
Filter and keep valuable data riemann / elasticsearch / influxdb
Generate realtime alerts riemann
Addictive dashboards kibana / grafana / riemann-dash

26. CHRONOS
CRON for mesos

28. Our workflow

29. Git flow
Jenkins DinD
Private Registry Docker
Doploy
Marathon & Chronos
Boot2Docker + Parallels

30. Don’t / Do

31. use tag "latest" on production
use explicit version tags for your
docker images
DON’T

32. use data containers for valuable data
storage layer may crash / change
easy to forget when cleaning unused
containers / images
DON’T

33. use shared storage (glusterfs, nfs, ...) for your codebase,
working dirs, config files or sessions
sloooooooooow
generate scary lock errors and timeouts
fscache crash full systems easily
DON’T

34. tag your custom images with the VCS commit hash
● makes your workflow better (same version tag in VCS
and images)
● prevent useless image rebuilds ( speedup deploys )
● easy way to know exactly which code version is running,
even if the tag was changed / deleted in the VCS
DO

35. avoid shared filesystems
● put all versioned data in containers
● use external object storage for user files (S3, ceph, swift ,...)
● use database / memcached / couchbase for sessions
● use templates to generate local config files (consul-template ,
confd)
DO

36. Use docker independant storage for critical data
● lvm is your friend (unless you use and saturate thin volume
metadata, use thin volumes with care)
● use storage plugins with docker >= 1.9.0 (convoy ?)
● redundant backups saves lives
DO

37. Mistaking
is
learning

38. - we tried to put the chicken in the egg
- generate config files to shared storage
- dns for discovery
- use mongo replicaset without sharding
- ...

39. Chronos Collectd Consul Consul-template Couchbase
Debian Docker Doploy ElasticSearch Fluentd Gearman
Go Grafana Haproxy HBase InfluxDB Java Jenkins
Kafka Kibana Marathon Mesos MongoDB Nodejs
OpenVpn PhalconPHP Redis Registrator Registry
Riemann Route53 S3 Scala Sentry Spark Statsd
Symfony Zookeeper ...

40. Switch from registrator to mesos-consul
Remove the lasts SPOFs
Move user files to S3
Use the docker storage/network plugins
XDCC
Improve logs & metrics filtering and alerting

41. 2015
Other mesos/docker stories
IPROFS
A large scale php/
drupal worldwide
social app for
“Institut français”
ARTE.TV
We are migrating
all their vod and
svod services to
mesos/docker
(java apps)
VIXNS
A mesos/docker
cluster is collecting
all logs and metrics
from hundreds of
servers

42. Conclusion

43. Questions?
@john_iw2 @vixns