Everything as Code principles and processes we used at Hepsiburada from Workstation Provisioning to Infrastructure. Learnings from our production experience
Agenda:
- Infrastructure As Code
- Workstation Provisioning
- Pipelines as Code
- Observability As Code
- ADRs
- Test Driven Everything
2. TECHNOLOGY
About Us
SERHAN GÜNEY
● Full-Stack Developer with 8 years of experience
● Currently working in Hepsiburada Marketplace
team as Senior Full-Stack Developer
https://www.linkedin.com/in/serhanguney/
serhangny@gmail.com
https://gitlab.com/serhan
ERDEM ERBAŞ
● Full-Stack developer with 2 years of experience
● Currently working in Hepsiburada Marketplace
team as Full-Stack Developer
https://www.linkedin.com/in/erdemerbas/
erdemerbas12699@gmail.com
5. TECHNOLOGY
Agenda
❖ Introduction <= You are here now
➢ Why Everything As Code
➢ Automation
➢ Gitops
➢ Code Review
➢ Testing
➢ Immutability
➢ Disposability
❖ Workstation Provisioning
❖ Infrastructure As Code
❖ Pipelines as Code
❖ Observability As Code
❖ ADRs
❖ Test Driven Everything
E
6. TECHNOLOGY
The idea behind the Everything as Code
concept is that infrastructure codified in a
declarative specification and treated like
application code such that they follow the same
software development lifecycle practices.
E
7. TECHNOLOGY Why Everything As Code
❖ Automation
❖ GitOps
❖ Code Review
❖ Testing
❖ Immutability
❖ Disposability
E
8. TECHNOLOGY Automation
❖ Higher production rates
❖ Increased productivity
❖ More efficient use of materials
❖ Better product quality
E
9. TECHNOLOGY GitOps
❖ Git as the source of truth
❖ Versioned CI/CD on top of declarative infrastructure
❖ Immutable deployment
❖ Stable and reproducible rollbacks
E
10. TECHNOLOGY Code Review
❖ Coding standards compliance
❖ Team cohesion
❖ Teaching and sharing knowledge
❖ Higher code quality
E
11. TECHNOLOGY Testing
❖ Testing verifies that the system meets the different requirements including,
functional, performance, reliability, security, usability and so on.
❖ Testing validates that the system being developed is what the user needs
E
12. TECHNOLOGY Immutability
❖ Build components to an exact set of specifications
❖ If a change to a specification is required, then a whole new set is
provisioned based on the updated requirements, and the previous is taken
out of service
❖ It is always possible to go back to the desired state
E
13. TECHNOLOGY Disposability
❖ Don’t Fix it, Throw it Away!
❖ It can be compared and contrasted with an application's disposability, one
of the characteristics of the 12 factor app
E
14. TECHNOLOGY
Agenda
❖ Introduction
❖ Workstation Provisioning <= You are here now
➢ What is workstation provisioning
➢ Onboarding
➢ Windows
➢ Linux
➢ Benefits
❖ Infrastructure As Code
❖ Pipelines as Code
❖ Observability As Code
❖ ADRs
❖ Test Driven Everything
S
16. TECHNOLOGY Onboarding
❖ Preconfigured workstation
❖ Instead of following a guideline just press start
❖ Configuration drift
❖ Time effective
❖ Before/After
S
18. TECHNOLOGY
Puppet
➢ Puppet is a desired state based configuration management automation
tool.
➢ Includes its own declarative language to describe system configuration
➢ Puppet supports all powershell commands(Powershell DSC)
E
19. TECHNOLOGY
Windows Provisioning
❖ Package
➢ Use chocolatey for default
package provider
➢ Taking advantage of powershell
desired state configuration
➢ Use specific versions(to avoid
breaking changes)
E
26. TECHNOLOGY
Multiple Boxes
❖ Differences of boxes are as it is in the image
➢ Seperate box for each business vertical
❖ In Common they all have
➢ required docker containers to ease local testing and debugging
➢ Git config to specify user
➢ ssh config to ease connection to build,qa and prod envs
E
27. TECHNOLOGY
Image Bakery vs CM
❖ Immutables like ansible should be in image bakery.
❖ Mutables like docker version should be configured by CM
E
29. TECHNOLOGY
Agenda
❖ Introduction
❖ Workstation Provisioning
❖ Infrastructure As Code <= You are here now
➢ Build Infra
➢ Environment Infra
❖ Pipelines As Code
❖ Observability As Code
❖ ADRs
❖ TDE
E
31. TECHNOLOGY
Infrastructure as Code
❖ Both On Premise
❖ Everything is a Docker Container
❖ Nginx as reverse-proxy
❖ No app-specific published ports
➢ Internal DNS
➢ Taking advantage of docker swarm mode SDN
E
33. TECHNOLOGY
Build - Environment Infra
❖ Separation of Concerns
❖ Build generates artifacts - Environment runs artifacts
❖ Reduces complexity
❖ Different networks and docker swarms
➢ Build swarm,
➢ Environment swarms(eg. qa, prod)
S
34. TECHNOLOGY
Agenda
❖ Workstation Provisioning
❖ Infrastructure As Code
➢ Build Infra <= You are here now
■ Bootstrap
■ Gitlab
■ GoCD
■ Docker Registry (Harbour)
■ Nexus
➢ Environment Infra
❖ Pipelines As Code
❖ Observability As Code
❖ ADRs
❖ TDE
S
36. TECHNOLOGY
Build Infra
❖ Where everything gets built
❖ Artifacts and Images are pushed to appropriate registries
➢ eg. docker registry, nexus
❖ Automated provisioning of build infra via bootstrap.sh
➢ cloud or on premise
❖ Uses Docker swarm mode for orchestration
S
37. TECHNOLOGY
Bootstrap
❖ Bootstrapping is only necessary when the build infrastructure is
provisioned for the first time
❖ Bootstrapping will install barebone build infra services
➢ Gitlab(with groups and project)
➢ GoCD
➢ Docker Registry , etc
❖ Just invoke the `bootstrap.sh` and chill
❖ Can be disposible `dispose.sh` and cry
S
38. TECHNOLOGY
Bootstrap
❖ bootstrap.sh creates base-images required during Bootstrap
➢ eg. alpine, curl, golang
❖ Solves dependencies with health checks
❖ Applicable even to devbox
➢ for testing purposes
❖ Two different configurations
➢ One for barebone bootstrap image
➢ Specialized one for the environment
S
39. TECHNOLOGY
Gitlab
❖ Our preferred VCS
❖ Holds all our code and configuration
❖ Initialization steps are done by bootstrapper
❖ Configuration
E
41. TECHNOLOGY
GoCD
❖ Our preferred CI/CD tool
❖ Pull based
❖ Agent based
➢ Capability based task scheduling
➢ Only docker capability(Only one agent to rule them all)
❖ Pipelines As Code
S
42. TECHNOLOGY
GoCD
❖ Base Build Infra pipelines are created by bootstrapper
❖ Updates itself via docker volume
❖ Slack Integration
❖ Pipeline as Code
❖ User Authentication
S
43. TECHNOLOGY
Docker Registry
❖ Holds all Docker images
❖ Initially used private docker registry
❖ Recently Harbor is used as container registry
➢ Registry authorization
E
44. TECHNOLOGY
Nexus
❖ Stores component type artifacts
❖ Keeping it internal helps you overcome external network issues
❖ Vagrant Box
➢ Raw repository
➢ Index.json (hyperlink)
❖ Apt package plug-in
E
45. TECHNOLOGY
Agenda
❖ Introduction
❖ Workstation Provisioning
❖ Infrastructure As Code
➢ Build Infra
➢ Environment Infra <= You are here now
■ Nginx
■ Consul
■ ElasticSearch
■ RabbitMQ
❖ Pipelines As Code
❖ Observability As Code
❖ ADRs
❖ TDE
E
46. TECHNOLOGY
Environment Infra
❖ Where Build Infra artifacts get to run
❖ Artifacts and Images are pulled from appropriate registries
➢ eg. docker registry, nexus
❖ Build once, run anywhere
➢ Configures image artifacts wrt environments
❖ Uses swarm as container scheduler
➢ 260 containers in total
➢ Will be replaced by k8s w/ istio
❖ Deployments are managed by constraints
E
47. TECHNOLOGY
Nginx
❖ High-performance HTTP server and reverse proxy
❖ Deploys on all nodes
❖ Creates all external overlay network definitions on deployment
❖ Configured via nginx.conf
➢ regexp to match wildcards
S
49. TECHNOLOGY
Consul
❖ Tool for service discovery and configuration
❖ Clustered
❖ Used for dynamic service configuration data and feature toggles
❖ Service discovery as RabbitMQ cluster peer discovery mechanisms
➢ RabbitMQ Cluster with HAProxy and Consul @Hepsiburada Tech
Blog
S
51. TECHNOLOGY
RabbitMQ
❖ The most widely deployed open source message broker.
❖ Uses consul based rabbitmq clustering
❖ Migrating RabbitMQ in a high traffic setup @Hepsiburada Tech Blog
❖ Manages plugins via enabled plugin configuration
❖ Can be configured via env variables or rabbitmq.conf
E
52. TECHNOLOGY
Environment Infra Also Includes
❖ Redis HA
❖ MongoDB
❖ Varnish
❖ ELK
❖ Prometheus
❖ Grafana
❖ Alert Manager
❖ Exporters
❖ Jaeger
Observability as Code
E
53. TECHNOLOGY
Agenda
❖ Introduction
❖ Workstation Provisioning
❖ Infrastructure as Code
❖ Pipelines as Code <= You are here now
➢ Definition of PaC
➢ Template Oriented Pipelines
➢ Examples
➢ Configuration
❖ Observability As Code
❖ ADRs
❖ TDE
S
54. TECHNOLOGY
Pipelines As Code
❖ Pipeline config is stored in VCS either with your application code or in
a separate repository
❖ Pipeline config repos are referenced at GoCD
❖ Application repos may contain .gocd.yml file
S
55. TECHNOLOGY
Template-based Pipelines
❖ Templates are stored at GoCD
❖ Docker eases templating
➢ same templates for all EaC,
➢ e.g. Golang API, React App, .Net Core API, ...
❖ Enables DSL-like pipeline definitions
➢ build-and-push,
➢ retag-and-push,
➢ auto-deploy,
➢ manual-deploy,
➢ deploy-prod, …
S
58. TECHNOLOGY
Agenda
❖ Workstation Provisioning
❖ ADRs
❖ Infrastructure As Code
❖ Pipelines as Code
❖ Observability As Code <= You are here now
➢ ELK
➢ Grafana
➢ Prometheus
➢ Tracing
❖ ADRs
❖ TDE
S
59. TECHNOLOGY
Observability As Code
❖ Where Build and Environment Infra are observed
❖ Resides with Environment Infra Code
❖ Three pillars of observability
➢ Logging
➢ Metrics
➢ Tracing
❖ Alerts
S
60. TECHNOLOGY
Logging
❖ ELK
❖ Logs shipped to ELK
➢ App logging
➢ Infra logging
❖ Watcher
❖ GELF for dockerized apps
❖ FileBeats for legacy apps
❖ Logstash pipeline
➢ grok
➢ gelf
S
66. TECHNOLOGY
Prometheus
❖ Holds all infra and app metrics
❖ Uses metrics exposed via exporters
❖ Constraints and docker volume utilized to persist data
❖ Generic and Environment based rules
❖ Slack Integration
➢ Alert Manager
S
74. TECHNOLOGY
ADRs
❖ An architectural decision (AD) is a software design choice that addresses
a significant requirement.
❖ MarkDown syntax is used for documents.
❖ ADR cli utilities
E
80. TECHNOLOGY
Future Roadmap
❖ Machine Provisioning
➢ Packer based images used in build/env infra
➢ SysAdmins manually provisions using this image
➢ Will be replaced by Terraform
❖ K8s w/ Istio migration
➢ Will be done after migrating the remaining legacy apps
❖ Secret Management