Wyobraź sobie, że w twojej aplikacji zachodzą jakieś zmiany (domain eventy). Chcielibyśmy te zmiany wystawić na zewnątrz, żebyśmy mogli na ich podstawie robić sobie raporty, read modele, sagi, synchronizować dane. Czy to zadanie okaże się być trudne czy proste, jeśli użyjemy bazy danych SQL. Co zyskaliśmy dzięki temu, że używam RDBMS/SQL a co utraciliśmy, być może, bezpowrotnie. W tej prezentacji opowiem wam jak chciałem zbudować pewną funkcjonalność dla biblioteki Rails Event Store, dlaczego okazało być się to trudniejsze niż myślałem, o modelu MVCC w PostgreSQL, czy jest sposób, żeby go obejść i uzyskać emulację trybu READ UNCOMMITTED. A może możnaby do całego problemu podejśc zupełnie inaczej i podłączyć się pod Write-Ahead-Log (WAL) i wygrać świat w ten sposób? Pokażę też jak moim zdaniem, korzystając z dokładnie tych samych konceptów, które stoją za Event Sourcingiem i bazami danych moglibyśmy budować API, tak bym za każdym razem pisząc integrację z serwisem X nie musiał się zastanawiać czy jego autorzy rozumieją pojęcie idempotent czy nie. Albo jak moglibyśmy osiągnąć prostotę dzięki używaniu Convergent Replicated Data Types (CRDT). Być może jako community stać nas na więcej niż REST nad CRUDem. Zastanowimy się, czy sprzedawcy SQLa zlasowali nam mózgi, sprawili, że zapomnieliśmy o najprostszym sposobie, który może działać i wprowadzili nas w maliny, w których aktualnie się znajdujemy. A może sami jesteśmy sobie winni? TLDR: Czy nasze aplikacje nie mogłyby działać tak jak pod spodem działają bazy danych? Czy to wszystko musi być takie ciężkie i skomplikowane jeśli chcemy mieć mikro-serwisy, zwłaszcza w małym zespole, który niekoniecznie lubi dostawiać 5 bazę danych do stacku technologicznego.

1. “
popuść wodze fantazji,
chcemy żeby ludzie na
sali mieli prawdziwy
segfault,
1

2. Czy sprzedawcy
SQLowych baz
danych nas
oszukali?
@pankowecki

3. Rails
EventStore
Ruby + Rails + SQL Event Store
3

4. EventStore? What?
Publish/Write Events
Read Events
4

5. Publish
event = OrderPlaced.new(data: {
order_id: 1,
order_data: "sample",
festival_id: "b2d506fd-409d-4ec7"
})
event_store.publish_event(
event,
stream_name: "order_1"
)
5

6. Usecases
◂ Event Sourcing
◂ Just a technical log
6

7. Target apps
◂ Greenfields
◂ CQRS
◂ DDD
◂ Event Sourcing
◂ Legacy apps
◂ Most of my life
7

8. 1.
Stream

9. Publish to a stream!
event = OrderPlaced.new(data: {
order_id: 1,
order_data: "sample",
festival_id: "b2d506fd-409d-4ec7"
})
event_store.publish_event(
event,
stream_name: "order_1",
expected_position: 0,
)
9

10. Stream = Named sequence of events. order_1
10
1
OrderPlacedcustom
erId:452,
totalPrice:887.21,
2
OrderPaidpaym
entGatew
ay:PayU
3
OrderShipped
postalService:DHL

11. SQL
ID StreamName Position Type Data
1 order_1 0 OrderPlaced ...
10 order_1 1 OrderShipped ...
23 order_1 2 OrderPaid ...
11

12. What if…?
2- 100
concurrent writes
to the same stream

13. It depends!
Every IT consultant, ever!
13

14. Concurrency level depends on use-cases
Event Sourcing
Optimistic or pessimistic
lock.
1 concurrent write to the
same stream.
Entity state depends on
previous events
Technical Log
Unlimited concurrency
on writes to the same
stream.
Independent events.
14

15. Event Sourcing
ID StreamName Position Type Data
1 order_1 0 OrderPlaced ...
10 order_1 1 OrderShipped ...
23 order_1 2 OrderPaid ...
15

16. Technical Log
ID StreamName Position Type Data
1 Wrocław NULL OrderPlaced ...
10 Wrocław NULL OrderPlaced ...
11 Wrocław NULL OrderPaid ...
16

17. 1.
EventStore
as
Queue

18. EventStore as Queue
18
205
FriendInvited
friend_id: 1000004
204
UserRegistered
fb_id: 1000004
203
OrderPaid
order_id: 2f5b
202
OrderPlaced
order_id: 2f5b

19. fck MQ

20. All events = Global Stream
ID StreamName Position Type Data
1 global NULL OrderPlaced ...
2 global NULL OrderShipped ...
3 global NULL OrderPaid ...
20

21. 21

22. 22

23. 2 solutions (that I know of) ...
Linearize all writes!
No transactions or short
transactions.
Transactions/Commits
occur one-by-one.
Global, defined order all
events (across all streams)
Workaround
But… how?
23

24. 1.
0. linearize writes

25. 25

26. All events = Global Stream (linearized writes)
ID StreamName Position Type Data
1 global 0 OrderPlaced ...
2 global 1 OrderShipped ...
3 global 2 OrderPaid ...
26

27. Target apps
◂ Greenfields
◂ CQRS
◂ DDD
◂ Event Sourcing
◂ Legacy apps
◂ Most of my life
27

28. Workarounds
(dragons)

29. 1.
1. Postgres logical
replication

30. WAL
Write-Ahead Logging
DBs already know how to sync
between primary node and replicas...
Postgres logical
replication
30

31. 31
INSERT INTO data(data) VALUES('3');
SELECT * FROM pg_logical_slot_peek_changes('regression_slot', NULL, NULL);
location | xid | data
-----------+-----+-----------------------------------------------
0/16E09C0 | 690 | BEGIN 690
0/16E09C0 | 690 | table public.data: INSERT: id[integer]:3 data[text]:'3'
0/16E0B90 | 690 | COMMIT 690
(3 rows)

32. “
Logical decoding takes the database’s write-ahead
log (WAL), and gives us access to row-level change
events: every time a row in a table is inserted,
updated or deleted, that’s an event.
Those events are grouped by transaction, and
appear in the order in which they were committed
to the database. Aborted/rolled-back transactions
do not appear in the stream.
Thus, if you apply the change events in the same
order, you end up with an exact, transactionally
consistent copy of the database. 32

33. “
The Postgres logical decoding is well designed: it
even creates a consistent snapshot that is
coordinated with the change stream.
You can use this snapshot to make a point-in-time
copy of the entire database (without locking — you
can continue writing to the database while the
copy is being made),
and then use the change stream to get all writes
that happened since the snapshot.
33

34. Wow!
great success!
34

35. “ Before you can use logical decoding,
you must set wal_level to logical and
max_replication_slots to at least 1.
35

36. “ The output plugin must be written in C
using the Postgres extension
mechanism, and loaded into the
database server as a shared library.
This requires superuser privileges and
filesystem access on the database
server, so it’s not something to be
undertaken lightly
36

37. “ This is all replication-based !!!
What happens when the client
(replica) stops working?
37

38. Maybe not
so great!
Custom team?
DevOps?
Monitoring?
Hosted Postgres
Go for it!
38

39. 1.
2. xmin for the rescue

40. 40

41. 41
SELECT *
FROM event_store_events_in_streams
ORDER BY id ASC
WHERE stream = 'global'
AND
id > last_seen_id
AND
xmin < txid_snapshot_xmin(txid_current_snapshot())

42. 42
Thread1: [1] [3]
Thread2: [2 ]
Query: Q
OK!

43. 43
Thread1: [1] [2, 4]
Thread2: [3, ]
Query: Q
FAIL!

44. 1.
3. transaction_id
in record and sync
transaction by
transaction

45. TransID = txid_current()
ID TransID Type Data
1 3 OrderPlaced ...
2 3 OrderShipped ...
3 4 OrderPaid ...
45

46. 46
SELECT *
FROM event_store_events_in_streams
ORDER BY trans_id, id ASC
WHERE stream = 'global' AND
(
trans_id > last_trans_id
OR (
trans_id = last_trans_id AND
id > last_id
) AND
trans_id < txid_snapshot_xmin(txid_current_snapshot())

47. It works!
i think ;) but…
it waits for longest transaction
even if does not write events
47

48. 1.
4. Escape MVCC with
locks… seriously….

49. 49

50. beyond MVVC in PGSQL
There are a few ways in-progress transactions can
communicate and affect each other:
● Via a shared client application (of course)
● SEQUENCE (and SERIAL) updates happen
immediately, not at commit time
● advisory locking
● Normal row and table locking, but within the
rules of READ COMMITTED visibility
● UNIQUE and EXCLUSION constraints
50

51. 51
SELECT
pg_advisory_lock(0) as getGlobalLock,
nextval('id_seq') as c1,
currval('id_seq') as c2,
pg_advisory_xact_lock(currval('id_seq')) as eid,
setval('id_seq', currval('id_seq') + size-1),
pg_advisory_unlock(0) as releaseGlobalLock,
before inserting events...

52. 52SELECT pg_try_advisory_xact_lock_shared(1)

53. It works!
i think ;) but…
it waits for longest transaction
which writes events
Call me crazy? Maybe?
53

54. 1.
EventStore as
Queue
=
synchronize append
only table

55. Solutions summary
◂ linearizing writies
◂ unsuitable for cloud
◂ not working
◂ max delay: longest transaction
◂ max delay: longest transaction which
writes events
55

56. no kurwa...

57. gimme WAL, stupid!

58. Czy sprzedawcy
SQLowych baz
danych nas
oszukali?

59. ● Use presented
solutions
● 2-* DBs
● another
DB/EventStore

60. ●

61. datalog?

62. We (IT industry)
suck at
exchanging data

63. CDC
Change
Data
Capture

64. 64

65. 65

66. debezium.io

67. CRDT
Convergent
Replicated
Data
Types

68. “
MC-Sets resolve divergent histories for an
element by choosing the value which has
changed the most. You cannot delete an
element which is not present, and cannot
add an element which is already present.
MC-sets are compact and do the right
thing when changes to elements are
infrequent compared to the conflict
resolution window, but behave arbitrarily
when divergent histories each include
many changes. 68

69. “
Each element e is associated with an
integer n, implicitly assumed to be zero.
When n is even, the element is absent
from the set. When n is odd, the element
is present. To add an element to the set,
increment n from an even value by one; to
remove an element, increment n from an
odd value by one. To merge sets, take
each element and choose the maximum
value of n from each history.
69

70. “
{
'type': 'mc-set',
'e': [
['a', 1],
['b', 2],
['c', 3]
]
70

71. 71
Thanks!
Any questions?
You can find us at
◂ @pankowecki
◂ @arkency

72. Credits
◂ https://www.confluent.io/blog/bottled-water-real-time-integration-of-postgre
sql-and-kafka/
◂ https://www.postgresql.org/docs/10/static/logicaldecoding.html
◂ https://www.slideshare.net/GrokkingVN/grokking-techtalk-20-postgresql-int
ernals-101
◂ https://stackoverflow.com/questions/33646012/postgresql-transaction-isolatio
n-read-uncommitted
◂ https://stackoverflow.com/questions/49201826/is-postgresl-serial-guaranteei
ng-no-gaps-within-single-insert-statement
◂ https://github.com/aphyr/meangirls
◂
72