Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Meet Spilo, Zalando’s HIGH-AVAILABLE POSTGRESQL CLUSTER - Feike Steenbergen

883 Aufrufe

Veröffentlicht am

In recent years Zalando has adopted a decentralized setup for applications and databases. This has impacted our database engineers by transferring responsibility to small teams, each of which manages its own infrastructure. Decentralization is great for team autonomy, but can present challenges in terms of how to easily manage lots of PostgreSQL clusters. That’s why our team created Spilo: an open source HA-cluster (highly available PostgreSQL cluster). This talk will show how Spilo simplifies Postgres cluster management while preserving team autonomy. By building upon streaming replication, Spilo provides a set of clusters that require no human interaction for many administration tasks and most failure scenarios; takes care of managing the number of servers (adding and removing them); and creates backups. It implements our own version of Patroni (https://github.com/zalando/patroni): a process, derived from Compose’s Governor, that governs the Postgres cluster (promoting and demoting) and updates information in etcd (the distributed consensus key/value store created by CoreOS). I’ll explore the architecture of Patroni implemented with Spilo; a live demo will show some failovers as they occur. Finally, I’ll show how Spilo combines Patroni with cloud infrastructure architecture components (for example, AWS), adding autoscaling to run a HA-cluster and allowing AWS power users to create a new HA-cluster with very little effort. Spilo relies upon STUPS, Zalando’s open-source platform as a service (PaaS) for enabling multiple, autonomous teams to use AWS while remaining audit-compliant. By using Spilo and STUPS together, our engineers can create a new HA-cluster with just a few commands. After attending this talk the audience will understand how they can also use Spilo, Patroni and STUPS to manage their Postgres clusters more efficiently while working autonomously.

Veröffentlicht in: Daten & Analysen
  • Als Erste(r) kommentieren

Meet Spilo, Zalando’s HIGH-AVAILABLE POSTGRESQL CLUSTER - Feike Steenbergen

  1. 1. Spilothe high-available PostgreSQL cluster Feike Steenbergen Zalando SE
  2. 2. • 15 EU countries • 3 fulfilment centers • 15+ million active customers • 2.2 billion € revenue 2014 Zalando
  3. 3. 150 000+ products
  4. 4. We are growing!
  5. 5. Our databases • > 150 production Postgresql databases • > 13.5 TB data • > 5 TB biggest DB • 400-1000+ write tps • > 2 DB failures/month
  6. 6. Zalando never sleeps
  7. 7. Infrastructure bottleneck ACID Team create alter deploy migrate failover upgrade 80+ teams
  8. 8. Radical Agility
  9. 9. Purpose
  10. 10. Autonomy
  11. 11. Mastery
  12. 12. Cloud • 2013: ZCloud • 2014: project Pequod • 2015: Let’s just use AWS…
  13. 13. Amazon abbreviations • AWS - amazon web services • EC2 - elastic compute cloud • ELB - elastic load balancer • RDS - relational DB service • CF - Cloud Formation • ASG - Auto Scaling Group
  14. 14. AWS • One account per team • Microservices • REST/OAuth2 • Deployment with Docker
  15. 15. Autonomous teams on AWS REST INTERNET
  16. 16. Autonomous teams • Team decides which product to build • … and which technologies to use • REST/OAuth2 mandatory • Team is responsible for its infrastructure
  17. 17. • Developers should take care of infrastructure • ..including production databases • On AWS! Databases?
  18. 18. Isn’t it dangerous? DBAs running with scissors, by Gavin M. Roy: https://www.flickr.com/photos/gavinmroy/4638958958
  19. 19. ACID team provides PostgreSQL trainings
  20. 20. What about failover?
  21. 21. • Detect the master failure • Elect a new master • Redirect clients Autofailover tasks
  22. 22. Autofailover issues • Discarded writes • Split-brain • False positives
  23. 23. RDS? • Support for PostgreSQL • Automatic failover • Most extensions • Automatic backups
  24. 24. RDS? • Vendor lock • No superuser • No untrusted languages • No logical decoding plugins • Costs
  25. 25. Spilo (სპილო)
  26. 26. Spilo does • Rapid deployment of PostgreSQL on AWS EC2 instances • Streaming replication with auto- failover
  27. 27. Spilo on AWS Spilo MASTER Spilo REPLICA Spilo REPLICA Master connection Application DB request ETCD cluster status update
  28. 28. Failover Spilo REPLICA Spilo REPLICA Master connection Application DB request ETCD cluster status update
  29. 29. Failover Spilo MASTER Spilo REPLICA Master connection Application DB request ETCD cluster status update NEW SPILO STARTS…
  30. 30. Failover Spilo MASTER Spilo REPLICA Master connection Application DB request ETCD cluster status update Spilo REPLICA
  31. 31. What is Spilo? c Patroni MASTER c Patroni REPLICA c Patroni REPLICA Auto-scaling group Auto-scaling group
  32. 32. Demo: Deploying Spilo • We use stups • First we define a template • We create a Cloud Formation Stack from this template
  33. 33. Patroni (პატრონი) • Handles new replicas and failover • Based on ideas and code of the Compose Governor • Open-source • Runs everywhere
  34. 34. Compose Governor idea ● Use etcd for failover decision ● Run etcd on every node ● Run 1 node with HAProxy + etcd
  35. 35. Distributed configuration systems • Fault tolerant • Reliably store small amounts of strongly-consistent data between distributed nodes • Good for storing the PostgreSQL cluster state
  36. 36. Distributed consensus LEADER CLIENT CLIENT CLIENT write request
  37. 37. Distributed consensus LEADER CLIENT CLIENT CLIENT write request LEADER
  38. 38. Cluster state in etcd $ export ETCD=172.17.0.2:4001 $ etcdctl -C $ETCD ls /service/ --recursive /service/dm /service/dm/optime /service/dm/optime/leader /service/dm/members /service/dm/members/postgresql_172_17_0_3 /service/dm/members/postgresql_172_17_0_4 /service/dm/members/postgresql_172_17_0_5 /service/dm/initialize /service/dm/leader
  39. 39. Leader key • Points to the member key • Has a TTL, autoexpires • Acts as an exclusive lock • Only the leader can become the master
  40. 40. Leader TTL $ http http://$ETCD/v2/keys/service/dm/leader ... { "action": "get", "node": { "createdIndex": 8, "expiration": "2015-11-20T09:56:43.59367 "key": "/service/dm/leader", "modifiedIndex": 85, "ttl": 22, "value": "postgresql_172_17_0_3" } }
  41. 41. Member key $ etcdctl -C $ETCD get /service/dm/members/postgresql_172_17_0_5 { "conn_url": "postgres://un:pw@172.17.0.5:5432/po "api_url": "http://172.17.0.5:8008/patroni", "tags": {}, "state": "running", "role": "replica", "xlog_location": 67109176 }
  42. 42. Patroni Patroni MASTER REPLICA MASTER LB PostgreSQL connection API HealthCheck (master) Connection and API URL
  43. 43. Patroni Patroni MASTER REPLICA MASTER LB PostgreSQL connection API HealthCheck (master) Connection and API URL
  44. 44. Patroni Patroni MASTER REPLICA MASTER LB PostgreSQL connection API HealthCheck (master) Connection and API URL
  45. 45. Patroni Patroni MASTER REPLICA MASTER LB PostgreSQL connection API HealthCheck (master) Connection and API URL
  46. 46. Connection and API URL Patroni Patroni MASTER REPLICA MASTER LB REPLICA LB API HealthCheck (slave) PostgreSQL connection API HealthCheck (master)
  47. 47. Initialize key $ etcdctl -C $ETCD get /service/dm/initialize 6219169399948550171 • PostgreSQL cluster system ID • Created by the first node that joins the cluster • Nodes with different system ID are not allowed to join
  48. 48. Patroni modules ETCD ZOOKEEPER ABSTRACT DCS PostgreSQL REST API High availability Asynchronous executor Callbacks
  49. 49. Demo time! https://asciinema.org/a/2ttvu50yehjo2712s1w43udio
  50. 50. • Robust exception handling • Run long-running tasks (i.e. base backup in a separate thread) • ETCD + Zookeeper • Rest API Patroni improvements
  51. 51. Patroni improvements • Configurable replica imaging • Support for pg_rewind • patronictl • packaged: pip install patroni
  52. 52. Patroni improvements • Manual failover • Initialize from external cluster • Attach to already running PostgreSQL nodes • Tags (i.e. nofailover)
  53. 53. • Spilo: github.com/zalando/spilo spilo.readthedocs.org • Patroni: github.com/zalando/patroni patroni.readthedocs.org • Stups: github.com/zalando-stups/ stups.io • Feedback: @ekief Thank you!

×