Empfohlen
Weitere ähnliche Inhalte
Ähnlich wie consistency without borders
Ähnlich wie consistency without borders (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
consistency without borders
- 1. consistency without borders Peter Alvaro, Peter Bailis, Neil Conway, Joseph M. Hellerstein UC Berkeley
- 2. The transaction concept DEBIT_CREDIT: BEGIN_TRANSACTION; GET MESSAGE; EXTRACT ACCOUT_NUMBER, DELTA, TELLER, BRANCH FROM MESSAGE; FIND ACCOUNT(ACCOUT_NUMBER) IN DATA BASE; IF NOT_FOUND | ACCOUNT_BALANCE + DELTA < 0 THEN PUT NEGATIVE RESPONSE; ELSE DO; ACCOUNT_BALANCE = ACCOUNT_BALANCE + DELTA; POST HISTORY RECORD ON ACCOUNT (DELTA); CASH_DRAWER(TELLER) = CASH_DRAWER(TELLER) + DELTA; BRANCH_BALANCE(BRANCH) = BRANCH_BALANCE(BRANCH) + DELTA; PUT MESSAGE ('NEW BALANCE =' ACCOUNT_BALANCE); END; COMMIT;
- 3. The transaction concept DEBIT_CREDIT: BEGIN_TRANSACTION; GET MESSAGE; EXTRACT ACCOUT_NUMBER, DELTA, TELLER, BRANCH FROM MESSAGE; FIND ACCOUNT(ACCOUT_NUMBER) IN DATA BASE; IF NOT_FOUND | ACCOUNT_BALANCE + DELTA < 0 THEN PUT NEGATIVE RESPONSE; ELSE DO; ACCOUNT_BALANCE = ACCOUNT_BALANCE + DELTA; POST HISTORY RECORD ON ACCOUNT (DELTA); CASH_DRAWER(TELLER) = CASH_DRAWER(TELLER) + DELTA; BRANCH_BALANCE(BRANCH) = BRANCH_BALANCE(BRANCH) + DELTA; PUT MESSAGE ('NEW BALANCE =' ACCOUNT_BALANCE); END; COMMIT;
- 4. An application-level contract Write Read Application Opaque store Transactions
- 9. Research on consistency Write Read Application Opaque store Consistency models R1(X=1) R2(X=1) W1(X=2) W2(X=0) W1(X=1) W1(Y=2) R2(Y=2) R2(X=0)
- 10. Research on consistency Write Read Application Opaque store Consistency models Assert: balance > 0 causal? PRAM? delta? fork/join? red/blue? release? SC? (translation) R1(X=1) R2(X=1) W1(X=2) W2(X=0) W1(X=1) W1(Y=2) R2(Y=2) R2(X=0)
- 11. Meanwhile, in industry… Application Opaque store Custom solutions Write Read
- 12. Meanwhile, in industry… Application Opaque store Custom solutions Assert: balance > 0 (prayer) Write Read
- 13. Distributed consistency: staying relevant • Is this an important problem? • Is academia disconnected from reality? • OK, what now? Goal: help programmers write correct applications. Today: some promising approaches
- 14. Case study: a graph T1 T2 T4 T10 T3 T6 T5 T9 T7 T8 T11 T12 T13 T14
- 17. Problem: deadlock detection Task: Identify strongly-connected components Waits-for graph T1 T2 T4 T10 T3 T6 T5 T9 T7 T8 T11 T12 T13 T14
- 18. Problem: garbage collection Task: Identify nodes not reachable from Root. Root Refers-to graph T1 T2 T4 T10 T3 T6 T5 T9 T7 T8 T11 T12 T13 T14
- 19. T1 T2 T4 T10 T3 T6 T5 T9 T7 T8 T11 T12 T13 T14 Correctness Deadlock detection • Safety: No false positives- • Liveness: Identify all deadlocks Garbage collection • Safety: Never GC live memory! • Liveness: GC all orphaned memory T1 T2 T4 T10 T3 T6 T5 T9 T7 T8 T11 T12 T13 T14 ParLLon Root
- 20. Consistency at the extremes Storage Object Flow Language Application Linearizable key-value store? Custom solutions?
- 21. Consistency at the extremes Storage Object Flow Language Application Linearizable key-value store? Custom solutions? Efficient Correct
- 22. Consistency across the stack
- 23. Object-level consistency Capture semantics of data structures that • allow greater concurrency • maintain guarantees (e.g. convergence) Storage Object Flow Language Application
- 24. Insert Read Convergent data structure (e.g., Set CRDT) Object-level consistency Insert Read Commutativity Associativity Idempotence Reordering Batching Retry/duplication Tolerant to
- 25. Application Convergent data structures Object-level consistency ? ? GC Assert: No live nodes are reclaimed Assert: Graph replicas converge
- 27. Flow-level consistency Capture semantics of data in motion • Asynchronous dataflow model • component properties à system-wide guarantees Graph store Transaction manager Transitive closure Deadlock detector Confluent ConfluentConfluent
- 28. Flow-level consistency Order-insensitivity (confluence) output set = f(input set) { } { } =
- 29. Flow-level consistency Confluence is compositional output set = f Ÿ g(input set)
- 30. Flow-level consistency Confluence is compositional output set = f Ÿ g(input set)
- 31. Graph store Memory allocator Transitive closure Garbage collector Confluent Not Confluent Confluent Graph store Transaction manager Transitive closure Deadlock detector Confluent ConfluentConfluent Graph queries as dataflow
- 32. Graph store Memory allocator Transitive closure Garbage collector Confluent Not Confluent Confluent Graph store Transaction manager Transitive closure Deadlock detector Confluent ConfluentConfluent Graph queries as dataflow Confluent Coordinate here
- 33. Language-level consistency DSLs for distributed programming? • Capture consistency concerns in the type system Storage Object Flow Language Application
- 34. Language-level consistency CALM Theorem: Monotonic à confluent Conservative, syntactic test for confluence
- 35. Language-level consistency Deadlock detector Garbage collector
- 36. Language-level consistency Deadlock detector Garbage collector nonmonotonic
- 37. Storage Object Flow Language Application correct reusable efficient intuitive Where we’ve been; where we’re headed
- 38. Storage Object Flow Language Application correct reusable efficient intuitive Where we’ve been; where we’re headed
- 39. Storage Object Flow Language Application correct reusable efficient intuitive Where we’ve been; where we’re headed
- 40. Remember • Consistency is an application-level property • Correctness and performance are compatible • Meet programmers on their home turf • Build bridges!
- 41. Queries?
Hinweis der Redaktion
- 2 things happened: academia ran down the stack, creating new I/O-level models with no accompanying intuitive contract for apps.Industry ran up the stack, developing best practices for app-level consistency without support (or guarantees!) from infrastructure
- Recover common ground with programmers that was lost with the abandonment of xacts
- To guard against failures, we replicate.NB: asynchrony => replicas might not agree
- Very similar looking criteria (1 safe 1 live). Takes some work, even on a single site. But hard in our scenario: disorder => replica disagreement, partial failure => missing partitions
- recall “the developer’s conundrum.” essentially 3 choices: 1) SC, meaningful at app level,
- Ie, reorderability, batchability, tolerance to duplication / retryNow programmer must map from application invariants to object API (with richer semantics than read/write).
- All API calls to add() commute; no CC necessaryApproach: adjacency lists as setsHowever, not sufficient to synchronize GC.
- However, not sufficient to synchronize GC.Perhaps more importantly, not *compositional* -- what guarantees does my app – pieced together from many convergent objects – give?To reason compositionally, need guarantees about what comes OUT of my objects, and how it transits the app.
- We are interested in the properties of component *outputs* rather than just internal state. Hence we are interested in a different property: confluence.A confluent module behaves like a function from sets (of inputs) to sets (of outputs)
- Confluence is compositional: Composing confluent components yields a confluent dataflow
- All of these components are confluent! Composing confluent components yields a confluent dataflowBut annotations are burdensome
- M – a semantic property of code – implies confluenceAn appropriately constrained language provides a conservative syntactic test for M.
- M – a semantic property of code – implies confluenceAn appropriately constrained language provides a conservative syntactic test for M.
- Also note that a data-centric language give us the dataflow graph automatically, via dependencies (across LOC, modules, processes, nodes, etc)
- Consistency models and mechanisms are only useful if they help programmers write correct applications.