This document discusses the evolution of notonthehighstreet.com's infrastructure from a monolithic Ruby on Rails application hosted on over 150 physical and virtual servers, to a microservices architecture using Docker containers, Mesos for clustering, and Consul for service discovery and configuration management. The goals of the new architecture were to build a scalable, self-service infrastructure that allows for easy creation and management of new services. Key aspects of the implementation include using Docker to define service environments, Mesos for container orchestration, Consul as a configuration store, and ELK stack for logging. Ansible is used for configuration management and deploying services via integration with tools like Marathon and Jenkins.

1. Building a
future-proof
infrastructure
Henrique Rodrigues
notonthehighstreet.com

2. About notonthehighstreet.com
● UK's largest curated marketplace
● More than 5,000 partners, 170,000 products
● ~200 employees, ¼ tech
● 9 years of strong growth
● A great place to buy really nice, original,
custom products from small businesses

3. Back in the day
We regret nothing

4. Back in the day
● Single Ruby on Rails service (MonoNOTHS)
● ~150 servers, both physical and virtual
● Puppet, OpenStack, Rundeck
● One server, one function
● Continuous integration and continuous
delivery, via Jenkins

5. Microservices
A new challenge

6. Microservices: problems
● A lot more services
● Different requirements for different services
● Different stacks
● The need to scale: easier, larger, faster
● Maintenance overhead
● Can't replicate DevOps fast enough

7. Microservices: a solution
● Docker containers
● Small footprint
● Easy to create
● Easy to move around
● Too good to be true?

8. Microservices: Docker problems
● Not a silver bullet
● Docker just simplifies running services
● No configuration management
● No persistent data
● No traditional logging
● No orchestration

9. A new architecture
Building the dream

10. A new architecture: goals
● Scalable infrastructure
● Service containment
● Configuration management
● Log management
● Easy creation of new services
● Self service QA environments

11. A new architecture: kitty metaphor

12. A new architecture: implementation
● Docker
● Mesos
● Consul
● ELK stack
● AWS
● Ansible
● A bit of glue

13. A new architecture: Docker
● Developers can define the service
environment as part of the codebase
● Unified deployments for all services
● Immutable, no maintenance needed
● We try to run everything with Docker,
including infrastructure services

14. A new architecture: Mesos
● Clustering environment
● Knows how to “speak” Docker
● Supports different frameworks, schedulers
● Marathon for running services
● Chronos for cron-like tasks
● APIs and UIs

15. A new architecture: Consul
● Secure key/value store
● Single source of truth for services
● Consul Template watches it and creates
configuration files, used by containers
● Useful for all services, even infrastructure
● APIs and UIs

16. A new architecture: ELK stack
● Elasticsearch + Logstash + Kibana
● Services create log files in Logstash JSON
● Logstash knows what to do, stores them in
Elasticsearch
● Kibana knows how to visualise logs
● Anyone can create custom dashboards

17. A new architecture: AWS
● It's really scalable, if there's any doubt
● Same architecture for prod and QA, resilient
● “Unlimited” QA environments for developers
● Single AMI, infrastructure services run in
containers and are enabled only if needed
● Low maintenance hosts

18. A new architecture: Ansible
● Masterless orchestration tool
● Also does configuration management
● Integrates with AWS natively
● Interfaces with Marathon to deploy services
● Pushes configuration of services to Consul
● Jobs are triggered via Jenkins, usually

19. A new architecture: a bit of glue
● Traffic routing via NGINX and PowerDNS,
backed by Consul
● Registrator service, hooks into Docker
daemon and tells Consul when other
services come alive or die

20. A new architecture: scary overview
Ansible
AWS
EC2 instances
RDS, memcached, etc.
ELBs, etc.
Mesos slaves
Infrastructure services:
● Mesos master
● Mesos slave
● Logstash
● Elasticsearch
● Marathon
● Chronos
● etc.
Marathon
DockerDocker hub
Jenkins
Consul
ELK
Monitoring
Mesos master
Consul-template

21. Final thoughts
We have those too

22. Final thoughts
● 11 months from idea to production
● Bleeding edge technology, it's getting better
● Make developers a key part of the design
process, they're your users
● Hackdays are really important
● AWS can be costly: spot instances are nice

23. The end
Questions?