In this talk, Damien describes the infrastructure Nuxeo has built around Docker containers, which is mainly based on CoreOS and Docker, and how it provides a way to generically run applications not only on a single host, but across a whole cluster of hosts. The resulting architecture can be used to implement a PaaS approach for any application.
3. What is Nuxeo platform
• A platform to develop you own content apps
• Highly pluggable thru extension points
• Model you content with Nuxeo Studio
• Point and click UI to define your business model
• with hot-reload
4. Nuxeo tech. stack
• Java stack : Seam + JSF for Nuxeo DM and DAM
• Rest API to build your JS Apps
• Nuxeo Studio (Saas) to model your business
• Thru data model (doc types, schemas)
• Thru Automation chains
6. Goals
• Customers ask to host their app
• Provide quick access to test instances (onboarding)
• Provide onDemand dev or CI instances
7. Goals
• Customers ask to host their app
• Provide quick access to test instances (onboarding)
• Provide onDemand dev or CI instances
This is the nuxeo.io promise
9. Manual OnDemand
• Based on AWS CloudFormation
• Manually deployed
• Works but :
• not scalable
• manual admin & monitoring
• difficult upgrade process
11. AWS Automation
• Provisionning may be automated
• Working mockup (python, celery, boto) but…
• Strong adherence to AWS
• difficult to test
• slow provisionning (one EC2 instance per Nuxeo instance)
• upgrade process still difficult
• scaling out is difficult
• sensible to failures
13. Current challenge
• Build a scalable infrastructure
• Quick provisioning (less than a minute)
• Easy setup (no admin needed)
• Easy upgrade process
• Automated healing
• Easy to test
15. Quick provisioning
• We don’t have to start
a server
• No VM overhead
• Start time is Nuxeo
start time
16. Data free containers
• Docker container
doesn’t hold data
• Upgrading is just using
a new version of the
image
• Healing means killing
and restarting
Nuxeo v5.9.3
S3Postgres
17. Data free containers
• Docker container
doesn’t hold data
• Upgrading is just using
a new version of the
image
• Healing means killing
and restarting
S3Postgres
18. Data free containers
• Docker container
doesn’t hold data
• Upgrading is just using
a new version of the
image
• Healing means killing
and restarting
S3Postgres
Nuxeo v5.9.4
19. Several containers per server
• Scaling is just a matter
of starting several
containers
• But means infinite CPU
/ RAM and IO
Nuxeo.io server
io
container
NXIO-0001
io
container
NXIO-0002
io
container
NXIO-0003
io
container
NXIO-XXXX
…
20. Challenge status
• Build a scalable infrastructure
• Quick provisioning (less than a minute)
• Easy setup (no admin needed)
• Easy upgrade process
• Automated healing
• Easy to test
21. How to solve the
other issues ?
… with a little help from my friends
24. Docker is not sufficient
• Docker handles processes at the OS level
• On the cluster level we need a companion
• To load balance the processes
• To check the statuses of the processes
• We need a docker ps at the cluster level
25. Introducing CoreOS
• A minimal Linux distribution (small footprint)
• Designed to handle cluster
• Providing cluster tooling
• Docker
• etcd
• fleet
https://coreos.com/
26. etcd : service discovery
• etcd is
• small distributed key value store
• keys may have TTL
• using Raft protocol to elect cluster leader
• suitable for :
• shared configuration
• service discovery
27. fleet : clustered init system
• fleet is :
• an init system at the cluster level
• compatible with systemd unit files
• suitable for :
• launching docker containers
• distributing load across cluster
• get the status of the jobs
29. Challenge status
• Build a scalable infrastructure
• Quick provisioning (less than a minute)
• Easy setup (no admin needed)
• Easy upgrade process
• Automated healing
• Easy to test
30. What are the admin tasks ?
• Launch the fleetctl command
• Setup DNS and reverse proxies
• Link container to external services (S3, Postgres etc…)
31. DNS and reverse proxy
• docker start containers and link host port to container
port
• The cluster has to redirect calls to mycustomDNS.io to the
running container meaning knowing
• the ip of the host
• the port on the host
39. Introducing Gogeta
• Easy to start as a docker container
!
!
• Written in Go
• Available at : http://github.com/nuxeo/gogeta
/usr/bin/docker run --rm --name gogeta -p 7777:7777 nuxeo/gogeta
40. Application startup unit
!
[Unit]!
Description=Sample startup script!
!
[Service]!
ExecStartPre= Set starting status in etcd!
ExecStart= Launch docker container!
ExecStartPost= Wait container to be launched and set app location in etcd!
ExecStop= Stop docker container!
ExecStopPost= Set stopped status in etcd!
41. Linking services
• Docker recommend to use ambassadors to link services
• Service may be hosted on other servers or outside of the
cluster
• We’ve implemented dynamic ambassadors
• http://coreos.com/blog/docker-dynamic-ambassador-
powered-by-etcd/
45. Dynamic ambassador
• Very easy to start
!
• Uses https://github.com/nuxeo/etcd-netfw
docker run --rm --name postgres-amb -P nuxeo/service-amb postgres-service
46. Launching commands
• A manager web app
now launches
commands :
• sets up basic etcd
values (like domain to
service resolution)
• launches fleet
commands (start & stop
apps)
• polls statuses
47. Challenge status
• Build a scalable infrastructure
• Quick provisioning (less than a minute)
• Easy setup (no admin needed)
• Easy upgrade process
• Automated healing
• Easy to test
49. From nuxeo.io…
• We’ve built a cluster infrastructure that :
• knows how to deploy Docker containers (with fleet)
• Expose those containers thru domain name resolution (with
gogeta)
• Link dependent services to the container (with etcd-netfw)
50. From nuxeo.io…
• Given that a container should represent an application
• Running this apps means :
• Launch it with runtime parameters
• Link it to its dependent services
• Then we can build a metamodel around it
51. …
name: io-container!
version: 1.0!
params:!
- env_tech_id: !
- description: The id of the environment to run!
- type:string!
- clid:!
- description: a registered connect client ID!
- type:string!
!
buildimage:!
path: "./docker"!
!
dependencies:!
- postgres.service!
- s3.service!
!
52. … to arken
• to be able to run any application
!
• it will take care of :
• Submitting the model to the cluster
• Building the image if needed
• Launching the container
• Registering everything in etcd
arkenctl start iocontainer.ark —domain myapp.mycluster.io
53. Arken status
• We are extracting things from http://github.com/nuxeo/nuxeo.io
• http://github.com/arkenio
• home of gogeta and etcd
• provide cluster init instruction
• arken and arkenctl tools needs to be done
• new opensource projects (not in nuxeo repo)
• pull requests are welcome
54. Conclusion
• Docker helped us to enter the micro containers world
• Lots of new features : multi-repo, automatic /etc/hosts etc…
• CoreOS provided us cluster deployment
• Arken wants to provide a higher deployment model