Neo4j   High Availability  New Auto-ClusterMichael Hunger - @mesirii                            1
High Availability Cluster  ๏Neo4j Enterprise  ๏Master-Slave Replication  ๏read-scaling and fault-tolerance  ๏eventual cons...
3 Separate Concerns (I)๏Cluster Management  •   Members join/leave/heartbeat๏Failover  •   Master Election  • Distribution...
3 Separate Concerns (II)๏Replication  •synchronized id-generation  • distributed locks  • pull, push of transactions  • in...
Pre 1.9 - Zookeeper                  5
Pre 1.9๏Apache Zookeeper took care of concerns  •   Cluster Management      ‣new members register with ZK  •   Failover   ...
HA ClusterCoordinator              RO-                Coordinator                         Slave                       Mast...
Pre 1.9 - Problems๏Additional setup and operations of a separate   component๏unreliable operation / hiccups๏longterm stabi...
Post 1.9 -Neo4j Auto Cluster                 9
Replace Zookeeper!?๏Implement Multi-Paxos ourselves๏simple, testable code๏only covers  • cluster management,  • master ele...
HA Cluster             11
What is Paxos?๏reliable consensus making๏broadcasting๏works even with unreliable communication  •message lost  • delays, i...
What is Paxos?                 13
Implementation๏everything is a State Machines  • SM = stateless enums + context  • Message = type enum + payload  • State ...
Implementation (II)๏everything is a State Machines  •   use timeouts for reliability  • handle failing messages  • decoupl...
Implementation (II)๏Paxos (3 roles)                   Acceptor  •   Proposer-SM                 Paxos  • Acceptor-SM      ...
Multi-Paxos (happy path)                                                              Acceptor              Learner       ...
TIMEOUTMulti-Paxos (happy path)                        PROMISE                                                ACCEPT      ...
Acceptor State Machine                         19
Heartbeat State Machine                          20
Implementation (III)๏HA Implementation uses state machines as   infrastructure๏notifications via listeners๏piggyback heartb...
Multi-Paxos๏everything is a State Machines  •   use timeouts for reliability  • handle failing messages  • decouple networ...
Unit-Testing•   Mock Time    ‣fast running tests despite timeouts•   Mock Network    ‣simulate delays, failing messages   ...
Unit-Test-Example                    24
Setup   •Config        • Video        • Auto-Setup Script (Demo)                                     25
Thank You - Questions?                         26
Nächste SlideShare
Wird geladen in ...5
×

New Neo4j Auto HA Cluster

2,784

Published on

In this talk, Michael Hunger is going to shed some light over the new High Availability architecture for the popular Neo4j Graph Database. We are going to look at the different variants of the Paxos protocol, master failover strategies and cluster management state handling. This piece of infrastructure poses non-trivial challenges to distributed consensus-finding, an interesting session for anyone into scalable systems.

Published in: Technologie
0 Kommentare
4 Gefällt mir
Statistiken
Notizen
  • Hinterlassen Sie den ersten Kommentar

Keine Downloads
Views
Gesamtviews
2,784
Bei Slideshare
0
Aus Einbettungen
0
Anzahl an Einbettungen
2
Aktionen
Geteilt
0
Downloads
28
Kommentare
0
Gefällt mir
4
Einbettungen 0
No embeds

No notes for slide

New Neo4j Auto HA Cluster

  1. 1. Neo4j High Availability New Auto-ClusterMichael Hunger - @mesirii 1
  2. 2. High Availability Cluster ๏Neo4j Enterprise ๏Master-Slave Replication ๏read-scaling and fault-tolerance ๏eventual consistency • write to master (push_factor) • write to slaves 2
  3. 3. 3 Separate Concerns (I)๏Cluster Management • Members join/leave/heartbeat๏Failover • Master Election • Distribution of Master-Status 3
  4. 4. 3 Separate Concerns (II)๏Replication •synchronized id-generation • distributed locks • pull, push of transactions • initial store synchronization 4
  5. 5. Pre 1.9 - Zookeeper 5
  6. 6. Pre 1.9๏Apache Zookeeper took care of concerns • Cluster Management ‣new members register with ZK • Failover ‣ZK stores Master and last TX-Id ‣ZK uses ZAB to determine new Master and distribute information 6
  7. 7. HA ClusterCoordinator RO- Coordinator Slave Master Slave Slave Coordinator 7
  8. 8. Pre 1.9 - Problems๏Additional setup and operations of a separate component๏unreliable operation / hiccups๏longterm stability๏no dynamic reconfig of the ZK cluster important for cloud setup 8
  9. 9. Post 1.9 -Neo4j Auto Cluster 9
  10. 10. Replace Zookeeper!?๏Implement Multi-Paxos ourselves๏simple, testable code๏only covers • cluster management, • master election 10
  11. 11. HA Cluster 11
  12. 12. What is Paxos?๏reliable consensus making๏broadcasting๏works even with unreliable communication •message lost • delays, invalid order๏does not guarantee progress 12
  13. 13. What is Paxos? 13
  14. 14. Implementation๏everything is a State Machines • SM = stateless enums + context • Message = type enum + payload • State = enum instance • switch on msg-type, implement logic Transition = handle() messages, 14
  15. 15. Implementation (II)๏everything is a State Machines • use timeouts for reliability • handle failing messages • decouple network and time ‣for testability • listeners interact on messages with outside world, sync or async 15
  16. 16. Implementation (II)๏Paxos (3 roles) Acceptor • Proposer-SM Paxos • Acceptor-SM Proposer Learner • Learner-SM ClusterState๏Cluster • Heartbeat Heartbeat 16
  17. 17. Multi-Paxos (happy path) Acceptor Learner Proposer (2 * f + 1) PREPARE PREPARE TIMEOUT VALUE PROMISE MATCH OR REJECT NO MATCH ACCEPT MATCHES TIMEOUT PROMISE? CHECK , STORE STORE ACCEPTED VALUE RESPONSES OR IF QUORUM REJECTED NO MET, CANCEL TIMEOUT STORE ... VALUE LEARN OUT OF ORDER MSG HANDLING other DELIVER A VALUE IS Learner ALL VALID MISSING ATOMIC BC LEARN TIMEOUT WE STILL 17 LEARN TIMEOUT DONT KNOW
  18. 18. TIMEOUTMulti-Paxos (happy path) PROMISE ACCEPT ... MATCHES TIMEOUT PROMISE? CHECK , STORE STORE ACCEPTED VALUE RESPONSES OR IF QUORUM REJECTED NO MET, CANCEL TIMEOUT STORE VALUE LEARN OUT OF ORDER MSG HANDLING other DELIVER A VALUE IS Learner ALL VALID MISSING ATOMIC BC LEARN TIMEOUT WE STILL LEARN TIMEOUT DONT KNOW LEARN REQ LEARN TIMEOUT HAVE LEARN VALUE OR LEARN FAIL DONT KNOW 18
  19. 19. Acceptor State Machine 19
  20. 20. Heartbeat State Machine 20
  21. 21. Implementation (III)๏HA Implementation uses state machines as infrastructure๏notifications via listeners๏piggyback heartbeat on messages๏master election • (all - failed) have to agree • Paxos BC needs quorum of total 21
  22. 22. Multi-Paxos๏everything is a State Machines • use timeouts for reliability • handle failing messages • decouple network and time ‣for testability • listeners interact on messages with outside world, sync or async 22
  23. 23. Unit-Testing• Mock Time ‣fast running tests despite timeouts• Mock Network ‣simulate delays, failing messages 23
  24. 24. Unit-Test-Example 24
  25. 25. Setup •Config • Video • Auto-Setup Script (Demo) 25
  26. 26. Thank You - Questions? 26
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×