Applause's journey from a monolith to a microservice environment running on Mesosphere DC/OS by Christopher Gianelloni

1. Journey from monolith to microservices
Utilizing microservice patterns with monoliths
Chris Gianelloni @wolf31o2
1

2. In the beginning…
Applause had several ways to deploy and manage software.
• Custom system management tool (sysdeploy)
• Basically a SSH wrapper for manually created systems
• Custom Docker image management tool (Platypus)
• Standardized AMIs, built w/ Packer, including Docker daemon
• Services in Docker containers w/ configuration using SaltStack
• Provides A/B testing and health checks
• Packer + Chef + Terraform
• Packer + Chef to bake AMIs
• Terraform to deploy using ASGs
• Mesosphere DC/OS
• OSS orchestration for “Docker” containers
2

3. Typical “old school” configuration system, written completely
in-house, limited in capabilities, author has long-since
departed the company
• SSH wrapper to copy files and run commands
• No instance management
• No user management
• No rollback features
• No documentation
• Unfamiliar code base to everyone
• Unable to look up problems
sysdeploy
3

4. In-house microservice deployment and service management
• Leverages CloudFormation for infrastructure
• Template-based system
• INI-style configuration files
• Output lookups
• Leverages SaltStack for some configuration management
• Uses roles for service management
• Services in Docker containers
• Supports health checking
• Supports A/B deployments
• Supports manual rollback
• Tied to AWS
• Lots of ELBs
Platypus
4

5. Utilizes common, public, OSS tools
• Common tools with existing user bases and communities
• Basically “best of breed” tools
• Packer for AMIs
• Chef for installing and configuring software
• Terraform for deploying baked AMIs
• Plethora of documentation for each tool
• Chef Server optional
• Composable and reusable pieces
• Output lookups (Chef + Terraform)
Packer +
Chef +
Terraform
5

6. Mesos, Marathon, and Metronome (and more)
• Consolidated and unified platform
• Leverages common OSS technologies
• Standardized application and service management
• Health checking for services
• Supports single-shot, or scheduled tasks
• Service discovery
• Metrics and log collection
• Integrated data services
• Configuration rollbacks
• Canary deployments
• Universe packages
Mesosphere
DC/OS
6

7. 7
Applause chose DC/OS to leverage previous work while
also moving to a scalable system using open source
components. This frees the Platform Delivery team to
provide new capabilities to the Applause Hosting
Platform which provide for our business needs.
• Open source with a vibrant and active community
• Strong feature set around an integrated platform
• Ability to colocate diverse workloads
• Microservices
• Data services
• AI / Machine learning / Analytics
• Simple interfaces using API, CLI, and GUI
• Enterprise features and support
• Appreciation for memes
Why DC/OS?

8. 8
DC/OS Architecture
Software layer is where containers
execute to provide services. This
includes Marathon applications,
Metronome jobs, and Mesos
frameworks.
Platform layer is Mesosphere DC/OS
services execute, which run in the host
operating system.
Infrastructure layer provides the host
and operating system which hosts our
stack, such as Amazon Web Services.

9. 9
DC/OS Node Types
Master nodes host DC/OS services and
provide the orchestration layer, service
discovery, and administrative interfaces.
Public agent nodes are public facing
and contain API routing and load
balancing of incoming requests to
backend services. These are agent
nodes with a public role.
Private agent nodes are internal and
host all other services. Services
communicate via East-West load
balancing.

10. Applause
Hosting
Platform
10

11. Mesosphere
Universe
packages +
Application
services
11
Packages and services which provide base value to the
platform to be used by all Applause services:
• ecr-login - AWS Elastic Container Registry login process
• Provides and updates credentials for fetching images
• marathon-lb - North-South load balancer
• Provides ingress load balancing from public slaves to services
running in private slaves
• hdfs - Hadoop Distributed File System
• Provides shared storage for artifacts, logs, etc.
• Provides storage layer for AI/ML and analytics processes
• linkerd - HTTP proxy
• Provides service discovery and service mesh
• Provides East-West load balancing across private slaves
• kong - API gateway
• Provides API routing to specific endpoints
• spark - Data processing framework
• Provides processing framework for AI/ML and analytics

12. • Chef cookbook wrapping community cookbook: https://supermarket.chef.io/cookbooks/dcos
• Custom recipes
• Monitoring agent
• Docker Engine installation and configuration
• Enhanced Networking (ena) driver
• Logging aggregation agent
• System users via data bag
• DC/OS volumes (volume0, etc)
• DC/OS workdir configuration
• Cookbook “bake_time”
• Packer templates to create “shared” images
• Start from “official” CentOS base images
• Patch
• Reboot
• Remove old kernels
• Run Chef
• Cleanup
How do we build DC/OS?
12

13. Chef
wrapper
“secret
sauce”
13
Disable some Chef resources by modifying resources at
converge time:
# These are resources which need to be modified in the upstream dcos
# cookbook to prevent them from executing at bake time
[
{ template: '/usr/src/dcos/genconf/config.yaml' },
{ execute: 'dcos-genconf' },
{ file: '/usr/src/dcos/genconf/serve/dcos_install.sh' },
{ execute: 'preflight-check' },
{ execute: 'dcos_install' },
].each do |res|
ruby_block "action-nothing-#{res.keys.first}[#{res.values.first}]" do
block do
r = resources(res)
r.action([:nothing])
end
only_if { node['chef-applause-dcos']['bake_time'] }
end
end

14. Packer
template
14
Runs Chef wrapper cookbook to produce a shared AMI:
{
"type": "chef-solo",
"version": "13.8.5",
"run_list": ["recipe[chef-applause-base]", "recipe[chef-applause-dcos]"],
"json": {
"chef-applause-dcos": {
"bake_time": true,
"chef_config_actions": ["nothing"]
},
"dcos": {
"dcos_enterprise": "{{user `dcos_enterprise`}}",
"dcos_role": "nil",
"dcos_version": "{{user `dcos_version`}}"
}
},
"cookbook_paths": [ "chef/cookbooks" ],
"data_bags_path": "chef/pubkeys/data_bags"
},

15. • Terraform
• Derived from Mesosphere’s AWS CloudFormation templates
• Originally a 1:1 translation
• Evolved over time, more customizations
• VPC per cluster
• Masters have public addresses / ELB for discovery
• Private slaves have only internal addresses
• Public slaves are behind ALB
• Autoscaling Groups + Launch Configs
• One group per DC/OS role
• Launch Configs write out node-specific Chef configuration
• Executes Chef client in cloud-init at boot
• IAM instance profiles used
• One profile per DC/OS role
How do we deploy DC/OS?
15

16. Launch
Configs
template
16
Provide outputs from terraform to Chef client:
#cloud-config
write_files:
- content: |
{
"dcos": {
"config": {
"aws_region": "${region}",
"cluster_name": "${stack_name}",
"exhibitor_address": "${internal_elb_dns_name}",
"master_external_loadbalancer": "${public_elb_dns_name}",
"num_masters": "${num_masters}",
"s3_bucket": "${exhibitor_s3_bucket}",
"superuser_username": "${superuser_username}",
"superuser_password_hash": "${superuser_password_hash}"
},
"dcos_enterprise": ${dcos_enterprise},
"dcos_role": "${dcos_role}",
"dcos_version": "${dcos_version}"
}
}
path: |-
/etc/chef/client-config.json

17. • Chef wrapper changes
• Pull Request made
• Tested with ChefDK and Test Kitchen
• Merged to master
• Tested again
• Pushed to Chef server
• Packer job executed
• Creates AMIs in AWS accounts
• Terraform job executed
• Creates AWS resources
• IAM accounts, profiles, instance profiles
• VPC, subnets, security groups
• ASGs, ELBs/ALBs
• Creates terraform outputs
Development to deployment workflow for DC/OS
17

18. • Application terraform updated
• Databases, caches, storage buckets, etc
• Service repository updated
• Pull Request made
• Unit tests, integration tests
• Merge to master (or deployment branch)
• Unit tests
• Docker image
• Integration tests
• Code coverage
• Push image
• Service deployment / promotion to DC/OS
• Metronome
• Marathon
• Kong
Development to deployment workflow for Applause services
18

19. • Migrate more workloads from legacy hosting
• Data science
• Analytics
• Build and test
• Other products
• Integrate services with in-cluster resources
• Data services
• Migrate scheduled jobs to Metronome
• Chronos
• cron
• Applause platform
• Migrate long-running tasks to Metronome
• Kubernetes in-cluster
What now?
19

20. Check out our careers page:
https://www.applause.com/working-at-applause/We’re hiring
20

21. Questions?
21