Presented at DevOpsDays Boston. Over the past few years, massive open online courses (MOOCs) powered by Open edX have become wildly popular, bringing free or low-cost education to millions of students around the world. Such success, however, presents a slew of challenging problems in terms of providing a scalable, robust, and secure platform. At Appsembler, we offer customers a fully managed and supported Open edX stack, all the way from the frontend web application to the backend services like ElasticSearch, MySQL, and MongoDB. With so many moving parts, we have come to realize the value of a multi-container, microservices-oriented architecture using Docker. In contrast to a single-container deployment of the Open edX stack, a multi-container approach allows us to scale different services independently; improves robustness since we can simply spin up new copies of containers if they go down; and results in improved security through greater segmentation and isolation. In addition to discussing these benefits, we'll also cover how we're managing deployments using Kubernetes for orchestration and service discovery along with Google Cloud infrastructure. Nate Aune is a developer and entrepreneur with over 15 years of professional experience building highly scalable web applications. Nate is also the founder and CEO of Appsembler. Morgan Robertson is a DevOps Engineer at Appsembler with experience in Docker, Ansible, Python, and automation tools.

1. Scaling Open edX
with Kubernetes
DevOpsDays Boston
9.15.2015

2. Who we are
Nate Aune
Morgan Robertson

3. What we’ll cover
● Background -- Open edX
● Introducing Kubernetes
● Kubernetes concepts
● Scaling + resiliency
● Open edX on Kubernetes

4. Open edX background
● edX: non-profit founded by MIT and Harvard
● 500+ courses, 5M students learning on edX.org
● edX released Open edX in June 2013
● Stanford, MongoDB, Salesforce, Google, Microsoft,
McKinsey, Johnson & Johnson, Smithsonian

5. Open edX - a catalyst for innovation
212 Contributors
One of the fastest
growing open
source projects on
Github

6. Technical components
LMS/CMS (Django/Python)
Forum (Sinatra/Ruby)
User DB (MySQL)
Course DB (Mongo)
Tasks (Celery/RabbitMQ)
Caching (Memcache)
Proxy (Nginx)
Search (ElasticSearch)
Mapreduce (Hadoop)

7. Hosting infrastructure
S3 for serving:
● static assets
● grade downloads
● certificate downloads
● videos (for mobile)
● Load balancer
● Application server(s)
● Database server(s)
● Search server
● Utility server (tasks)
● Caching server
● Hadoop cluster

8. Typical scalable deployment of Open edX on AWS

9. Introducing Kubernetes
● Scheduling + orchestration layer for
containerized applications
● Abstracts your infrastructure
● Open source project by Google
● Production-ready as of July 2015

10. Kubernetes architecture

11. Kubernetes vs. the Docker triad
Kubernetes Swarm Compose Machine
Scheduling ✔ ✔
Service discovery ✔ ✅
Container scaling ✔ ✔
Machine provisioning ✅ ✔
Health checking ✔
Secret management ✔
Production-ready ✔

12. Kubernetes core concepts
● Pods
● Services
● Replication controllers

13. Pods
● Group of containers + volumes scheduled together
● Smallest deployable unit
● Containers share certain resources including network
stack

14. Services

15. Services
● Endpoint for a set of pods
● IP address, port, and label selectors
● Use round-robin routing to direct traffic to backend
pods

16. Services + Pods

17. Replication Controllers
● Manage pod lifecycles for a number of replicas
● Provide scaling + fault tolerance
● Use label selectors

18. Pods + Services + Replication Controllers

19. Scaling with Kubernetes
● Replication controllers scale pods
● Services provide a single endpoint for a group of
pods
● The Kubernetes master schedules pods across nodes

20. Resiliency with Kubernetes
● Replication controllers ensure a number of pods are
running
● Services provide load balancing
● Health checks allow bad pods to be ignored/removed

21. Open edX on Kubernetes
● Goals:
○ Multi-tenant
○ Scalable + resilient

22. The challenge

23. Architecture

24. Monitoring with Sysdig

25. Sysdig drill-down

26. Lessons learned
● Containers should be stateless
● Put initialization tasks into separate pods that run
once
● Services can be used to abstract non-containerized
components

27. Conclusion
● We’re still learning, but..
○ Kubernetes is a promising technology for
providing both scalability and resiliency

28. More info
Open edX - http://open.edx.org
Kubernetes - http://kubernetes.io
Google Container Engine - http://cloud.
google.com/container-engine

29. Thank you for your time!
Questions?
Slides: http://bit.ly/open-edx-kubernetes
nate@appsembler.com
morgan@appsembler.com
@appsembler