Based off of David Pick

1. @jna_sh
The LISP in the
Machine

2. </speaker>
@jna_sh
@jna_sh

3. </content>
<content>
@jna_sh
Braintree

4. </content>
<content>
@jna_sh
Thanks for having me!

5. @jna_sh

6. @jna_sh

7. </content>
<content>
@jna_sh
A story about Data
at Braintree

8. </content>
<content>
@jna_sh
Why me?

9. </content>
<content>
@jna_sh
I’m not a systems
engineer

10. </content>
<content>
@jna_sh
I don’t write
Clojure

11. </content>
<content>
@jna_sh
I do write
Haskell.

12. </content>
<content>
@jna_sh
…and Node.js

13. </content>
<content>
@jna_sh
I love cool tech.

14. </content>
<content>
@jna_sh
Especially FP.

15. </content>
<content>
@jna_sh
Braintree = Rails

16. </content>
<content>
@jna_sh
Go

17. </content>
<content>
@jna_sh
Haskell!!!1one

18. </content>
<content>
@jna_sh
Clojure

19. </content>
<content>
@jna_sh
Payments processor

20. </content>
<content>
@jna_sh
Uber | AirBnb | Minecraft

21. </content>
<content>
@jna_sh
1 million rides a
day.

22. </content>
<content>
@jna_sh
Vast amounts of
data.

23. </content>
<content>
@jna_sh
Powered by Clojure

24. </content>
<content>
@jna_sh
Building a real-
time data pipeline.

25. @jna_sh

26. </content>
<content>
@jna_sh
Once upon a time…

27. </content>
<content>
@jna_sh
Primary DB

28. @jna_sh
Data Warehouse

29. </content>
<content>
@jna_sh
Backup /
duplication

30. </content>
<content>
@jna_sh
Source of truth

31. </content>
<content>
@jna_sh
Low impact to live
transactions.

32. @jna_sh
Amazon Redshift

33. </content>
<content>
@jna_sh
Batch Processes

34. </content>
<content>
@jna_sh
updatedAt/createdAT

35. </content>
<content>
@jna_sh
Batch updates:

36. </content>
<content>
@jna_sh
Slow.

37. </content>
<content>
@jna_sh
Unpredictable.

38. </content>
<content>
@jna_sh
Can’t track
deletes.

39. </content>
<content>
@jna_sh
Missed updates.

40. </content>
<content>
@jna_sh
Burden of Knowledge

41. </content>
<content>
@jna_sh
Search

42. @jna_sh

43. @jna_sh
Architecture
Primary DB Batch process Redshift data
warehouse
Transaction
search

44. </content>
<content>
@jna_sh
PGQ

45. @jna_sh
Queuing system on
top of Postgres

46. </content>
<content>
@jna_sh
ACID

47. </content>
<content>
@jna_sh
Doesn’t block live
transactions.

48. </content>
<content>
@jna_sh
Elasticsearch

49. </content>
<content>
@jna_sh
Another place to
sync data.

50. @jna_sh
Architecture
Primary DB PGQ Redshift data
warehouse
Elasticsearch

51. </content>
<content>
@jna_sh
PGQ prioritises DB
integrity

52. </content>
<content>
@jna_sh
Potential for lost
messages.

53. </content>
<content>
@jna_sh
Redshift &
Elasticsearch…

54. </content>
<content>
@jna_sh
…Fall over quite
often.

55. </content>
<content>
@jna_sh
Where do we persist
our messages?

56. </content>
<content>
@jna_sh
Enter Kafka.

57. </content>
<content>
@jna_sh
Apache | LinkedIn

58. </content>
<content>
@jna_sh
PubSub messaging
system

59. </content>
<content>
@jna_sh
Cluster of Kafka
nodes

60. </content>
<content>
@jna_sh
Multi-produce
multi-consume

61. </content>
<content>
@jna_sh
Zookeeper - Failure
states

62. </content>
<content>
@jna_sh
Topics

63. </content>
<content>
@jna_sh
Categories of
Messages

64. </content>
<content>
@jna_sh
Partitions

65. </content>
<content>
@jna_sh
Split by Machine

66. </content>
<content>
@jna_sh
No rules of topic
writing.

67. </content>
<content>
@jna_sh
Messages given an
offset.

68. </content>
<content>
@jna_sh
Deleted after time
set by user.

69. </content>
<content>
@jna_sh
Kafka properties:

70. </content>
<content>
@jna_sh
Replays

71. </content>
<content>
@jna_sh
Strongly ordered

72. </content>
<content>
@jna_sh
But only by
partition.

73. @jna_sh
Architecture
Primary DB
PGQ
Kafka Redshift data
warehouse
Elasticsearch

74. </content>
<content>
@jna_sh
Gateway

75. </content>
<content>
@jna_sh
Databases sharded
by Merchant

76. </content>
<content>
@jna_sh
Partition fed by
shard

77. </content>
<content>
@jna_sh
Strong ordering per
merchant

78. </content>
<content>
@jna_sh
Categorise by
semantics of data

79. </content>
<content>
@jna_sh
Redshift needs
shape of data

80. </content>
<content>
@jna_sh
Elasticsearch needs
meaning

81. </content>
<content>
@jna_sh
topics = datastream | eventstream

82. </content>
<content>
@jna_sh
Cool! Job done.

83. </content>
<content>
@jna_sh
Time to build it.

84. </content>
<content>
@jna_sh
Clojure.

85. </content>
<content>
@jna_sh
Why Clojure?

86. </content>
<content>
@jna_sh
Rails

87. </content>
<content>
@jna_sh
No Lisp

88. </content>
<content>
@jna_sh
No JVM

89. </content>
<content>
@jna_sh
Because reasons

90. </content>
<content>
@jna_sh
JVM

91. </content>
<content>
@jna_sh
Kafka, Zookeeper,
Elasticsearch are JVM

92. </content>
<content>
@jna_sh
Laziness

93. </content>
<content>
@jna_sh
Infinite Lazy
Streams

94. </content>
<content>
@jna_sh
Testing

95. </content>
<content>
@jna_sh
Concurrency

96. </content>
<content>
@jna_sh
Threads

97. </content>
<content>
@jna_sh
Goroutines

98. </content>
<content>
@jna_sh
Actors

99. </content>
<content>
@jna_sh
Built-in shutdown
logic

100. </content>
<content>
@jna_sh
Status of actors

101. </content>
<content>
@jna_sh
Single merchant

102. </content>
<content>
@jna_sh
Offload work to
Kafka

103. </content>
<content>
@jna_sh
Elasticsearch
aliases

104. </content>
<content>
@jna_sh
What did we learn?

105. </content>
<content>
@jna_sh
Garbage Collection

106. </content>
<content>
@jna_sh
Boo hiss

107. </content>
<content>
@jna_sh
Keep it small

108. </content>
<content>
@jna_sh
G1GC

109. </content>
<content>
@jna_sh
Heap size is
important

110. </content>
<content>
@jna_sh
Smaller = Better

111. </content>
<content>
@jna_sh
Monitor all the
things

112. </content>
<content>
@jna_sh
Don’t use deault
configs

113. </content>
<content>
@jna_sh
Use a good
concurrency model

114. </content>
<content>
@jna_sh
Future gains

115. </content>
<content>
@jna_sh
Real time source of
truth

116. </content>
<content>
@jna_sh
Real time fraud
monitoring

117. </content>
<content>
@jna_sh
Real time reports

118. </content>
<content>
@jna_sh
Thank you!

119. </speaker>
@jna_sh
@jna_sh

120. @jna_sh
Thanks_ Subtitle