Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
The computer science behind a
modern distributed data store
Max Neunhöffer
OSDC, Berlin, 12 June 2018
www.arangodb.com
Overview
Topics
Resilience and Consensus
Sorting
Log-structured Merge Trees
Hybrid Logical Clocks
Distributed ACID Transac...
Overview
Topics
Resilience and Consensus
Sorting
Log-structured Merge Trees
Hybrid Logical Clocks
Distributed ACID Transac...
Resilience and Consensus
The Problem
A modern data store is distributed,
Resilience and Consensus
The Problem
A modern data store is distributed, because it needs to scale out and/or
be resilient.
Resilience and Consensus
The Problem
A modern data store is distributed, because it needs to scale out and/or
be resilient...
Resilience and Consensus
The Problem
A modern data store is distributed, because it needs to scale out and/or
be resilient...
Resilience and Consensus
The Problem
A modern data store is distributed, because it needs to scale out and/or
be resilient...
Resilience and Consensus
The Problem
A modern data store is distributed, because it needs to scale out and/or
be resilient...
Paxos and Raft
Traditionally, one uses the Paxos Consensus Protocol (1998).
More recently, Raft (2013) has been proposed.
...
Paxos and Raft
Traditionally, one uses the Paxos Consensus Protocol (1998).
More recently, Raft (2013) has been proposed.
...
Paxos and Raft
Traditionally, one uses the Paxos Consensus Protocol (1998).
More recently, Raft (2013) has been proposed.
...
Paxos and Raft
Traditionally, one uses the Paxos Consensus Protocol (1998).
More recently, Raft (2013) has been proposed.
...
Paxos and Raft
Traditionally, one uses the Paxos Consensus Protocol (1998).
More recently, Raft (2013) has been proposed.
...
Raft in a slide
An odd number of servers each keep a persisted log of events.
Raft in a slide
An odd number of servers each keep a persisted log of events.
Everything is replicated to everybody.
Raft in a slide
An odd number of servers each keep a persisted log of events.
Everything is replicated to everybody.
They ...
Raft in a slide
An odd number of servers each keep a persisted log of events.
Everything is replicated to everybody.
They ...
Raft in a slide
An odd number of servers each keep a persisted log of events.
Everything is replicated to everybody.
They ...
Raft in a slide
An odd number of servers each keep a persisted log of events.
Everything is replicated to everybody.
They ...
Raft in a slide
An odd number of servers each keep a persisted log of events.
Everything is replicated to everybody.
They ...
Raft in a slide
An odd number of servers each keep a persisted log of events.
Everything is replicated to everybody.
They ...
Raft demo
file:///home/neunhoef/talks/talks/public/Presentation/
2018-06-12-OSDC-Berlin/raft/raft.github.io/raftscope/inde...
Sorting
The Problem
Data stores need indexes. In practice, we need to sort things.
Sorting
The Problem
Data stores need indexes. In practice, we need to sort things.
Most published algorithms are rubbish o...
Sorting
The Problem
Data stores need indexes. In practice, we need to sort things.
Most published algorithms are rubbish o...
Sorting
The Problem
Data stores need indexes. In practice, we need to sort things.
Most published algorithms are rubbish o...
Idea for a parallel sorting algorithm: Merge Sort
Min−Heap:
sorted
merged
Idea for a parallel sorting algorithm: Merge Sort
Min−Heap:
sorted
merged
Nearly all comparisons hit the L2 cache!
Log structured merge trees (LSM-trees)
The Problem
People rightfully expect from a data store, that it
can hold more data ...
Log structured merge trees (LSM-trees)
The Problem
People rightfully expect from a data store, that it
can hold more data ...
Log structured merge trees (LSM-trees)
(Source: http://www.benstopford.com/2015/02/14/log-structured-merge-trees/, Author:...
Log structured merge trees (LSM-trees)
LSM-trees — summary
writes first go into memtables,
all files are sorted and immutabl...
Hybrid Logical Clocks (HLC)
The Problem
Clocks in different nodes of distributed systems are not in sync.
Hybrid Logical Clocks (HLC)
The Problem
Clocks in different nodes of distributed systems are not in sync.
general relativit...
Hybrid Logical Clocks (HLC)
The Problem
Clocks in different nodes of distributed systems are not in sync.
general relativit...
Hybrid Logical Clocks (HLC)
The Problem
Clocks in different nodes of distributed systems are not in sync.
general relativit...
Hybrid Logical Clocks (HLC)
The Idea
Every computer has a local clock, and we use NTP to synchronize.
Hybrid Logical Clocks (HLC)
The Idea
Every computer has a local clock, and we use NTP to synchronize.
If two events on diff...
Hybrid Logical Clocks (HLC)
The Idea
Every computer has a local clock, and we use NTP to synchronize.
If two events on diff...
Hybrid Logical Clocks (HLC)
The Idea
Every computer has a local clock, and we use NTP to synchronize.
If two events on diff...
Distributed ACID Transactions
Atomic either happens in its entirety or not at all
Consistent reading sees a consistent sta...
Distributed ACID Transactions
Atomic either happens in its entirety or not at all
Consistent reading sees a consistent sta...
Distributed ACID Transactions
The Problem
In a distributed system:
How to make sure, that all nodes agree on whether
the t...
Distributed ACID Transactions
The Problem
In a distributed system:
How to make sure, that all nodes agree on whether
the t...
Distributed ACID Transactions
The Problem
In a distributed system:
How to make sure, that all nodes agree on whether
the t...
Distributed ACID Transactions
The Problem
In a distributed system:
How to make sure, that all nodes agree on whether
the t...
Distributed ACID Transactions
The Problem
In a distributed system:
How to make sure, that all nodes agree on whether
the t...
Distributed ACID Transactions
WITHOUT
Distributed databases without ACID transactions:
ArangoDB, BigTable, Couchbase, Data...
Distributed ACID Transactions
WITHOUT
Distributed databases without ACID transactions:
ArangoDB, BigTable, Couchbase, Data...
Distributed ACID Transactions
Basic Idea
Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of
a data i...
Distributed ACID Transactions
Basic Idea
Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of
a data i...
Distributed ACID Transactions
Basic Idea
Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of
a data i...
Distributed ACID Transactions
Basic Idea
Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of
a data i...
Distributed ACID Transactions
Basic Idea
Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of
a data i...
Thanks
Thanks
and YES: We are hiring, too!
Links
http://the-paper-trail.org/blog/consensus-protocols-paxos
https://raft.github.io
https://en.wikipedia.org/wiki/Merge...
Nächste SlideShare
Wird geladen in …5
×

OSDC 2018 | The Computer science behind a modern distributed data store by Max Neunhöffer

10 Aufrufe

Veröffentlicht am

What we see in the modern data store world is a race between different approaches to achieve a distributed and resilient storage of data. Most applications need a stateful layer which holds the data. There are at least three necessary ingredients which are everything else than trivial to combine and of course even more challenging when heading for an acceptable performance. Over the past years there has been significant progress in respect in both the science and practical implementations of such data stores. In his talk Max Neunhoeffer will introduce the audience to some of the needed ingredients, address the difficulties of their interplay and show four modern approaches of distributed open-source data stores.
Topics are:
– Challenges in developing a distributed, resilient data store
– Consensus, distributed transactions, distributed query optimization and execution
– The inner workings of ArangoDB, Cassandra, Cockroach and RethinkDB
The talk will touch complex and difficult computer science, but will at the same time be accessible to and enjoyable by a wide range of developers.

Veröffentlicht in: Software
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

OSDC 2018 | The Computer science behind a modern distributed data store by Max Neunhöffer

  1. 1. The computer science behind a modern distributed data store Max Neunhöffer OSDC, Berlin, 12 June 2018 www.arangodb.com
  2. 2. Overview Topics Resilience and Consensus Sorting Log-structured Merge Trees Hybrid Logical Clocks Distributed ACID Transactions
  3. 3. Overview Topics Resilience and Consensus Sorting Log-structured Merge Trees Hybrid Logical Clocks Distributed ACID Transactions Bottom line: You need CompSci to implement a modern data store
  4. 4. Resilience and Consensus The Problem A modern data store is distributed,
  5. 5. Resilience and Consensus The Problem A modern data store is distributed, because it needs to scale out and/or be resilient.
  6. 6. Resilience and Consensus The Problem A modern data store is distributed, because it needs to scale out and/or be resilient. Different parts of the system need to agree on things.
  7. 7. Resilience and Consensus The Problem A modern data store is distributed, because it needs to scale out and/or be resilient. Different parts of the system need to agree on things. Consensus is the art to achieve this as well as possible in software. This is relatively easy, if things are good, but very hard, if:
  8. 8. Resilience and Consensus The Problem A modern data store is distributed, because it needs to scale out and/or be resilient. Different parts of the system need to agree on things. Consensus is the art to achieve this as well as possible in software. This is relatively easy, if things are good, but very hard, if: the network has outages, the network has dropped or delayed or duplicated packets, disks fail (and come back with corrupt data), machines fail (and come back with old data), racks fail (and come back with or without data).
  9. 9. Resilience and Consensus The Problem A modern data store is distributed, because it needs to scale out and/or be resilient. Different parts of the system need to agree on things. Consensus is the art to achieve this as well as possible in software. This is relatively easy, if things are good, but very hard, if: the network has outages, the network has dropped or delayed or duplicated packets, disks fail (and come back with corrupt data), machines fail (and come back with old data), racks fail (and come back with or without data). (And we have not even talked about malicious attacks and enemy action.)
  10. 10. Paxos and Raft Traditionally, one uses the Paxos Consensus Protocol (1998). More recently, Raft (2013) has been proposed. Paxos is a challenge to understand and to implement efficiently. Various variants exist. Raft is designed to be understandable.
  11. 11. Paxos and Raft Traditionally, one uses the Paxos Consensus Protocol (1998). More recently, Raft (2013) has been proposed. Paxos is a challenge to understand and to implement efficiently. Various variants exist. Raft is designed to be understandable. My advice: First try to understand Paxos for some time (do not implement it!), then enjoy the beauty of Raft,
  12. 12. Paxos and Raft Traditionally, one uses the Paxos Consensus Protocol (1998). More recently, Raft (2013) has been proposed. Paxos is a challenge to understand and to implement efficiently. Various variants exist. Raft is designed to be understandable. My advice: First try to understand Paxos for some time (do not implement it!), then enjoy the beauty of Raft, but do not implement it either!
  13. 13. Paxos and Raft Traditionally, one uses the Paxos Consensus Protocol (1998). More recently, Raft (2013) has been proposed. Paxos is a challenge to understand and to implement efficiently. Various variants exist. Raft is designed to be understandable. My advice: First try to understand Paxos for some time (do not implement it!), then enjoy the beauty of Raft, but do not implement it either! Use some battle-tested implementation you trust!
  14. 14. Paxos and Raft Traditionally, one uses the Paxos Consensus Protocol (1998). More recently, Raft (2013) has been proposed. Paxos is a challenge to understand and to implement efficiently. Various variants exist. Raft is designed to be understandable. My advice: First try to understand Paxos for some time (do not implement it!), then enjoy the beauty of Raft, but do not implement it either! Use some battle-tested implementation you trust! But most importantly: DO NOT TRY TO INVENT YOUR OWN!
  15. 15. Raft in a slide An odd number of servers each keep a persisted log of events.
  16. 16. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody.
  17. 17. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority.
  18. 18. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority. Only the leader may append to the replicated log.
  19. 19. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority. Only the leader may append to the replicated log. An append only counts when a majority has persisted and confirmed it.
  20. 20. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority. Only the leader may append to the replicated log. An append only counts when a majority has persisted and confirmed it. Very smart logic to ensure a unique leader and automatic recovery from failure.
  21. 21. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority. Only the leader may append to the replicated log. An append only counts when a majority has persisted and confirmed it. Very smart logic to ensure a unique leader and automatic recovery from failure. It is all a lot of fun to get right, but it is proven to work.
  22. 22. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority. Only the leader may append to the replicated log. An append only counts when a majority has persisted and confirmed it. Very smart logic to ensure a unique leader and automatic recovery from failure. It is all a lot of fun to get right, but it is proven to work. One puts a key/value store on top, the log contains the changes.
  23. 23. Raft demo file:///home/neunhoef/talks/talks/public/Presentation/ 2018-06-12-OSDC-Berlin/raft/raft.github.io/raftscope/index.html http://raft.github.io/raftscope/index.html (by Diego Ongaro)
  24. 24. Sorting The Problem Data stores need indexes. In practice, we need to sort things.
  25. 25. Sorting The Problem Data stores need indexes. In practice, we need to sort things. Most published algorithms are rubbish on modern hardware.
  26. 26. Sorting The Problem Data stores need indexes. In practice, we need to sort things. Most published algorithms are rubbish on modern hardware. The problem is no longer the comparison computations but the data movement .
  27. 27. Sorting The Problem Data stores need indexes. In practice, we need to sort things. Most published algorithms are rubbish on modern hardware. The problem is no longer the comparison computations but the data movement . Since the time when I was a kid and have played with an Apple IIe, compute power in one core has increased by ×20000 a single memory access by ×40 and now we have 32 cores in a CPU this means computation has outpaced memory access by ×1280!
  28. 28. Idea for a parallel sorting algorithm: Merge Sort Min−Heap: sorted merged
  29. 29. Idea for a parallel sorting algorithm: Merge Sort Min−Heap: sorted merged Nearly all comparisons hit the L2 cache!
  30. 30. Log structured merge trees (LSM-trees) The Problem People rightfully expect from a data store, that it can hold more data than the available RAM, works well with SSDs and spinning rust, allows fast bulk inserts into large data sets, and provides fast reads in a hot set that fits into RAM.
  31. 31. Log structured merge trees (LSM-trees) The Problem People rightfully expect from a data store, that it can hold more data than the available RAM, works well with SSDs and spinning rust, allows fast bulk inserts into large data sets, and provides fast reads in a hot set that fits into RAM. Traditional B-tree based structures often fail to deliver with the last 2.
  32. 32. Log structured merge trees (LSM-trees) (Source: http://www.benstopford.com/2015/02/14/log-structured-merge-trees/, Author: Ben Stopford, License: Creative Commons)
  33. 33. Log structured merge trees (LSM-trees) LSM-trees — summary writes first go into memtables, all files are sorted and immutable, compaction happens in the background, merge sort can be used, all writes use sequential I/O, Bloom filters or Cuckoo filters for fast reads, =⇒ good write throughput and reasonable read performance, used in ArangoDB, BigTable, Cassandra, HBase, InfluxDB, LevelDB, MongoDB, RocksDB, SQLite4 and WiredTiger, etc.
  34. 34. Hybrid Logical Clocks (HLC) The Problem Clocks in different nodes of distributed systems are not in sync.
  35. 35. Hybrid Logical Clocks (HLC) The Problem Clocks in different nodes of distributed systems are not in sync. general relativity poses fundamental obstructions to synchronizity, in practice, clock skew happens, Google can use atomic clocks, even with NTP (network time protocol) we have to live with ≈ 20ms.
  36. 36. Hybrid Logical Clocks (HLC) The Problem Clocks in different nodes of distributed systems are not in sync. general relativity poses fundamental obstructions to synchronizity, in practice, clock skew happens, Google can use atomic clocks, even with NTP (network time protocol) we have to live with ≈ 20ms. Therefore, we cannot compare time stamps from different nodes!
  37. 37. Hybrid Logical Clocks (HLC) The Problem Clocks in different nodes of distributed systems are not in sync. general relativity poses fundamental obstructions to synchronizity, in practice, clock skew happens, Google can use atomic clocks, even with NTP (network time protocol) we have to live with ≈ 20ms. Therefore, we cannot compare time stamps from different nodes! Why would this help? establish “happened after” relationship between events, e.g. for conflict resolution, log sorting, detecting network delays, time to live could be implemented easily.
  38. 38. Hybrid Logical Clocks (HLC) The Idea Every computer has a local clock, and we use NTP to synchronize.
  39. 39. Hybrid Logical Clocks (HLC) The Idea Every computer has a local clock, and we use NTP to synchronize. If two events on different machines are linked by causality, the cause should have a smaller time stamp than the effect.
  40. 40. Hybrid Logical Clocks (HLC) The Idea Every computer has a local clock, and we use NTP to synchronize. If two events on different machines are linked by causality, the cause should have a smaller time stamp than the effect. causality ⇐⇒ a message is sent Send a time stamp with every message. The HLC always returns a value > max(local clock, largest time stamp ever seen).
  41. 41. Hybrid Logical Clocks (HLC) The Idea Every computer has a local clock, and we use NTP to synchronize. If two events on different machines are linked by causality, the cause should have a smaller time stamp than the effect. causality ⇐⇒ a message is sent Send a time stamp with every message. The HLC always returns a value > max(local clock, largest time stamp ever seen). Causality is preserved, time can “catch up” with logical time eventually. http://muratbuffalo.blogspot.com.es/2014/07/hybrid-logical-clocks.html
  42. 42. Distributed ACID Transactions Atomic either happens in its entirety or not at all Consistent reading sees a consistent state, writing preser- vers consistency Isolated concurrent transactions do not see each other Durable committed writes are preserved after shut- down and crashes
  43. 43. Distributed ACID Transactions Atomic either happens in its entirety or not at all Consistent reading sees a consistent state, writing preser- vers consistency Isolated concurrent transactions do not see each other Durable committed writes are preserved after shut- down and crashes (All relatively doable when transactions happen one after another!)
  44. 44. Distributed ACID Transactions The Problem In a distributed system: How to make sure, that all nodes agree on whether the transaction has happened? (Atomicity)
  45. 45. Distributed ACID Transactions The Problem In a distributed system: How to make sure, that all nodes agree on whether the transaction has happened? (Atomicity) How to create a consistent snapshot across nodes? (Consistency)
  46. 46. Distributed ACID Transactions The Problem In a distributed system: How to make sure, that all nodes agree on whether the transaction has happened? (Atomicity) How to create a consistent snapshot across nodes? (Consistency) How to hide ongoing activities until commit? (Isolation)
  47. 47. Distributed ACID Transactions The Problem In a distributed system: How to make sure, that all nodes agree on whether the transaction has happened? (Atomicity) How to create a consistent snapshot across nodes? (Consistency) How to hide ongoing activities until commit? (Isolation) How to handle lost nodes? (Durability)
  48. 48. Distributed ACID Transactions The Problem In a distributed system: How to make sure, that all nodes agree on whether the transaction has happened? (Atomicity) How to create a consistent snapshot across nodes? (Consistency) How to hide ongoing activities until commit? (Isolation) How to handle lost nodes? (Durability) We have to take replication, resilience and failover into account.
  49. 49. Distributed ACID Transactions WITHOUT Distributed databases without ACID transactions: ArangoDB, BigTable, Couchbase, Datastax, Dynamo, Elastic, HBase, MongoDB, RethinkDB, Riak, and lots more . . . WITH Distributed databases with ACID transactions: (ArangoDB,) CockroachDB, FoundationDB, Spanner
  50. 50. Distributed ACID Transactions WITHOUT Distributed databases without ACID transactions: ArangoDB, BigTable, Couchbase, Datastax, Dynamo, Elastic, HBase, MongoDB, RethinkDB, Riak, and lots more . . . WITH Distributed databases with ACID transactions: (ArangoDB,) CockroachDB, FoundationDB, Spanner =⇒ Very few distributed engines promise ACID, because this is hard!
  51. 51. Distributed ACID Transactions Basic Idea Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of a data item are kept.
  52. 52. Distributed ACID Transactions Basic Idea Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of a data item are kept. Do writes and replication decentrally and distributed, without them becoming visible from other transactions.
  53. 53. Distributed ACID Transactions Basic Idea Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of a data item are kept. Do writes and replication decentrally and distributed, without them becoming visible from other transactions. Then have some place, where there is a switch, which decides when the transaction becomes visible.
  54. 54. Distributed ACID Transactions Basic Idea Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of a data item are kept. Do writes and replication decentrally and distributed, without them becoming visible from other transactions. Then have some place, where there is a switch, which decides when the transaction becomes visible. These “switches” need to be persisted somewhere (durability), scale out (no bottleneck for commit/abort), be replicated (no single point of failure), be resilient in case of fail-over (fault-tolerance).
  55. 55. Distributed ACID Transactions Basic Idea Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of a data item are kept. Do writes and replication decentrally and distributed, without them becoming visible from other transactions. Then have some place, where there is a switch, which decides when the transaction becomes visible. These “switches” need to be persisted somewhere (durability), scale out (no bottleneck for commit/abort), be replicated (no single point of failure), be resilient in case of fail-over (fault-tolerance). Transaction visibility needs to be implemented (MVCC), time stamps play a crucial role.
  56. 56. Thanks Thanks and YES: We are hiring, too!
  57. 57. Links http://the-paper-trail.org/blog/consensus-protocols-paxos https://raft.github.io https://en.wikipedia.org/wiki/Merge_sort http://www.benstopford.com/2015/02/14/log-structured-merge-trees/ http://muratbuffalo.blogspot.com.es/2014/07/hybrid-logical-clocks.html https://research.google.com/archive/spanner.html https://www.cockroachlabs.com/docs/v1.1/architecture/overview.html https://www.arangodb.com http://mesos.apache.org

×