Overview of Stampede.io

1. CoreOS + Digitial Ocean Meetup 9/8/14
Darren Shepherd

2. About Me – Darren
Shepherd
@ibuildthecloud, darren0
Fancy Title Engineer @ Citrix
Previously Cloud Architect @ GoDaddy 3 yrs
Building IaaS systems for past 5 yrs
Apache CloudStack – Committer
OpenStack
2 from scratch proprietary clouds
Stampede.io is the 4th orchestration platform
I’ve built

3. What is Stampede.io
Hybrid IaaS/Docker orchestration platform
Can run both VMs and Containers in a
consistent fashion
Share approaches where makes sense, but still
respects that Containers != VMs
Easy to install/upgrade and use
Tailored installation for CoreOS
Cattle.io is the more raw framework under the
hood

6. Boring Reason
Personal R&D project @ Citrix
Just me, my opinions, and a MacBook Pro (running Linux)
Loose upfront goals
Make IaaS somehow better
○ Apply 5 years of lessons learned
Include Docker/Containers
○ Shiny new tech is fun
Spent 6 months locked in a closet
Hacking – ~50k lines of code
Staring at wall – “How do I think containers can be useful?”

7. Quite Literally…
My home office is a
closet

8. UI
Developed by Vincent Fiduccia
(vincent99 on Github)
Node.js and Ember
Event driven using WebSockets

9. Either changing the world or world domination are
expected outcomes

10. Container Playbook
Containers are portable
Containers can run on a laptop, in a VM, or on
Bare metal
Bare metal becomes more attractive
Faster and cheaper
Bare metal + Container == world domination
Hypervisors, Virtualization, VMware, AWS, etc
are doomed

11. But there’s a problem
Containers are portable
Containers are a compute technology
Storage and Networking are not portable
EBS – Reliable storage w/ snapshots
VPC – IP addressing, firewall, L2

12. Nothing we can’t Architect
around
Ephemeral apps
Distributed storage
NoSQL - Cassandra
Architect for the cloud!

13. Darren’s brief history of EC2
2006 – Ephemeral only VMs
2008 – EBS
2009 – VPC
2010-2014 – Crazy exponential growth

14. EBS and VPC are essential
You can architect for the cloud
But many won’t
Amazon didn’t convince everyone to re-architect
for the cloud
Amazon supported legacy architectures
Containers are currently EC2 2006
Most people will continue to run on the
infrastructure they already have
Not a game changer

15. Containers won’t change the
infrastructure world unless we
tackle storage and networking

16. The theory
Container are portable because Linux is
ubiquitous
Great idea, why didn’t I think of that
We can build a portable EBS and VPC with
just Linux
Linux has the majority of the technology
needed, we just need to piece it together.

17. Stampede is about building a
portable cloud – compute,
storage, and networking

18. What does this mean?
Infrastructure providers only need to provide
Linux
Simple block storage
L3 connectivity
If VT-x/SVM is available, VMs can be launched
Stampede provides everything else
Stampede can be provided “as a Service”
Normalize the infrastructure market
Massive scale infrastructure provider is not needed

20. Default Stampede Deployment
CoreOS
Node 1
Controller
Agent
Libvirt
CoreOS
Node 2
Agent
Libvirt
CoreOS
Node 3
Agent
Libvirt
One node becomes
controller

21. Logical Components in
Controller
Deployed in single
process for
simplicity
Should scale for
clouds <50 servers
Controller
Database
Lock Manager
Event Bus
API Server
Process Server
Agent Server

22. Fully Distributed Controller

23. Stampede on Stampede

24. Digital Ocean Deployment
3 Etcd clusters w/ Fleet
Management Stack – 3 x 12GB
SFO1 Nodes – 100 x 2GB
NYC3 Nodes – 100 x 2GB

25. Tests
Sustained rate 6-7 containers per second
Not tuned for raw throughput (naively stupid
scheduler)
Gigantic batches
10000 containers in one API call (count=10000)
Restart everything during deployments
Agents, Servers, Database, ZooKeeper, etc
Just tried to break things

26. How Many Containers?

28. A lot – 127,884
127,884 Running Containers
~600 per VM
17 failures (0.01%)
~6 hours
Was not testing for throughput
Focus on reliability
Note: This was done with unmanaged
networking