Video: https://www.parleys.com/tutorial/life-beyond-illusion-present Summary: The idea of the present is an illusion. Everything we see, hear and feel is just an echo from the past. But this illusion has influenced us and the way we view the world in so many ways; from Newton’s physics with a linearly progressing timeline accruing absolute knowledge along the way to the von Neumann machine with its total ordering of instructions updating mutable state with full control of the “present”. But unfortunately this is not how the world works. There is no present, all we have is facts derived from the merging of multiple pasts. The truth is closer to Einstein’s physics where everything is relative to one’s perspective. As developers we need to wake up and break free from the perceived reality of living in a single globally consistent present. The advent of multicore and cloud computing architectures meant that most applications today are distributed systems—multiple cores separated by the memory bus or multiple nodes separated by the network—which puts a harsh end to this illusion. Facts travel at the speed of light (at best), which makes the distinction between past and perceived present even more apparent in a distributed system where latency is higher and where facts (messages) can get lost. The only way to design truly scalable and performant systems that can construct a sufficiently consistent view of history—and thereby our local “present”—is by treating time as a first class construct in our programming model and to model the present as facts derived from the merging of multiple concurrent pasts. In this talk we will explore what all this means to the design of our systems, how we need to view and model consistency, consensus, communication, history and behaviour, and look at some practical tools and techniques to bring it all together.

1. Life BEYOND
the ILLUSION
of PRESENT
Jonas Bonér
CTO Typesafe
@jboner

2. "Time is what prevents everything from happening at once."
- John Archibald Wheeler

3. Newton's
PHYSICS

4. THIS SIMPLIFIED MODEL IS
VERY APPEALING TO US

5. von Neumann
ARCHITECTURE

6. BACK THEN
LIFE WAS GOOD

7. THEN, ALONG CAME
CONCURRENCY
MADE LIFE MISERABLE

8. Jim Gray's
TRANSACTIONS
SAVE THE DAY

10. WELL, ALONG CAME
DISTRIBUTION
MADE LIFE MISERABLE, again...

11. But don't be surprised
UNFORTUNATELY,
THIS IS NOT
HOW THE WORLD WORKS

12. "The future is a function of the past."
- A. P. Robertson

13. "The (local) present is a merge function of
multiple concurrent pasts."
— Me

14. val newLocalPresent = observedPasts.
foldLeft(oldLocalPresent) { _ merge _ }

15. WE NEED TO EXPLICITLY MODEL THE
LOCAL PRESENT AS
FACTS DERIVED FROM THE
MERGING OF MULTIPLE
CONCURRENT PASTS

16. INFORMATION IS ALWAYS FROM THE PAST

17. THE TRUTH
IS CLOSER TO
EINSTEIN'S
PHYSICS

18. INFORMATION HAS
LATENCY

19. THE COST OF MAINTAINING
THIS ILLUSION IS INCREASED
CONTENTION &
COHERENCY

20. AS LATENCY GETS HIGHER, THE
ILLUSION CRACKS EVEN MORE

21. DISTRIBUTED SYSTEMS
EVERYWHERE

22. THE NETWORK IS RELIABLE...NAT

23. "If a tree falls in a forest and no one is around to hear it, does it make a
sound?"
— Charles Riborg Mann

24. Information CAN (and will) GET LOST

25. HOW DO WE DEAL WITH
INFORMATION LOSS
IN REAL LIFE?

26. WE USE A SIMPLE PROTOCOL OF
Confirm, Wait & Repeat

27. We fill in THE BLANKS

28. ...AND IF WE ARE WRONG, WE TAKE
COMPENSATING ACTIONAPOLOGY-ORIENTED PROGRAMMING - PAT HELLAND (IN MEMORIES, GUESSES, AND APOLOGIES)

29. The bottom line:
WE CAN'T FORCE THE WORLD INTO A
SINGLE GLOBALLY CONSISTENT
PRESENT

30. Should we just GIVE UP?

31. I BELIEVE THAT THERE IS A
PATH FORWARD

32. WE NEED TO TREAT
TIME AS A
FIRST CLASS CONSTRUCT

33. WHAT IS TIME, really?

34. TIME IS THE
SUCCESSION OF
CAUSALLY RELATED EVENTS

35. How can we
MANAGE
TIME?

36. Think in FACTS

37. What is a
FACT?

38. "A Fact is something that truly exists or happens: something that has
actual existence, a true piece of information."
— Merriam Webster

39. IMMUTABILITY
is a requirement

40. So, do variables
HAVE A PURPOSE IN LIFE?

41. "The assignment statement is the von Neumann bottleneck of
programming languages and keeps us thinking in word-at-a-time terms
in much the same way the computer's bottleneck does."
— John Backus (Turing Award lecture 1977)

42. MUTABLE STATE
NEEDS TO BE
CONTAINED

43. Ok, but how should we
MANAGE FACTS?

44. Functional
PROGRAMMING

45. Logic
PROGRAMMING

46. Dataflow
PROGRAMMING

47. NEVER
DELETE
FACTS

48. "When bookkeeping was done with clay tablets or paper and ink,
accountants developed some clear rules about good accounting
practices. One never alters the books; if an error is made, it is
annotated and a new compensating entry is made in the books. The
books are thus a complete history of the transactions of the business.
Update-in-place strikes many systems designers as a cardinal sin: it
violates traditional accounting practices that have been observed for
hundreds of years."
— Jim Gray (1981)

49. CRUD

50. "Database is a cache of a subset of the log."
— Pat Helland (2007)

51. Store facts in an
EVENT LOG

52. The log allows
TIME
TRAVEL

53. Can we REWRITE THE PAST?

54. Allows us to shift our focus from
DATA AT REST, to
DATA IN MOTION

55. Stream Processing

56. CONSTRUCTING A SUFFICIENTLY CONSISTENT
LOCAL PRESENT
MEANS EMPLOYING
CONSISTENCY MECHANISMS

57. Consistency
WHAT?
WHY?
WHEN?

58. WE NEED TO DECOMPOSE THE SYSTEM USING
CONSISTENCY BOUNDARIES

59. INSIDE DATA: OUR CURRENT PRESENT
OUTSIDE DATA: BLAST FROM THE PAST
BETWEEN SERVICES: HOPE FOR THE FUTURE
— PAT HELLAND (DATA ON THE INSIDE VS DATA ON THE OUTSIDE)

60. MicroSERVICE

61. AGGREGATE Root

62. WITHIN THE CONSISTENCY BOUNDARY

63. EVENT
Sourcing

64. BETWEEN THE
Consistency
Boundaries
IT'S A ZOO

65. Decoupling in
TIME / SPACE

66. STRONG
CONSISTENCY
The wrong default

67. Here, we are living in the
LOOMING SHADOW OF
IMPOSSIBILITY
THEOREMS

68. FLPCONSENSUS IS IMPOSSIBLE

69. PROTOCOLS CLIMB THE LADDER OF KNOWLEDGE
Cɸ: Common Knowledge (infinite number of i)
Eiɸ: (Everyone knows * i) ɸ
E3ɸ: (Everyone knows * 3) ɸ
E2ɸ: Everyone knows Everyone knows ɸ
E1ɸ: Everyone knows ɸ
Sɸ: Someone knows ɸ

70. COMMON KNOWLEDGE
IS NOT ATTAINABLE VIA PROTOCOL- JOSEPH HALPERN

71. CAPCONSISTENCY IS IMPOSSIBLE

72. Dissecting
CAP

73. "The first principle of successful scalability is to batter the consistency
mechanisms down to a minimum."
– James Hamilton

74. EVENTUAL
CONSISTENCY
What does it really mean?

75. Tracking TIME is tracking CAUSALITY

76. RELYING ON
TIMESTAMPS
IS A BAD IDEA

77. Instead, rely on
LOGICAL TIME

78. Lamport
CLOCKS
GLOBAL CAUSAL ORDERING BETWEEN

79. Vector
CLOCKSPARTIAL CAUSAL ORDERING BETWEEN EVENTS

80. Causal
CONSISTENCY

81. What
CONSISTENCY
DO YOU REALLY NEED AND
when?

82. ACID 2.0ASSOCIATIVE
COMMUTATIVE
IDEMPOTENT
DISTRIBUTED

83. CONFLICT-FREE REPLICATED DATA TYPES

84. DISORDERLY PROGRAMMING
CALM THEOREM

85. WE ARE JUST GETTING STARTED

86. WE HAVE A LONG ROAD AHEAD OF US...

88. Thanks
FOR LISTENING

89. Life BEYOND
the ILLUSION
of PRESENT
Jonas Bonér
CTO Typesafe
@jboner