At Booking.com, we have a constant flow of events coming from various applications and internal subsystems. This critical data needs to be stored for real-time, medium and long term analysis. Events are schema-less, making it difficult to use standard analysis tools.This presentation will explain how we built a storage and analysis solution based on Riak. The talk will cover: data aggregation and serialization, Riak configuration, solutions for lowering the network usage, and finally, how Riak's advanced features are used to perform real-time data crunching on the cluster nodes.

1. Event storage and real-time analysis at
Booking.com with Riak
Damien Krotkine

2. Damien Krotkine
• Software Engineer at Booking.com
• github.com/dams
• @damsieboy
• dkrotkine

3. • 800,000 room nights reserved per day
WE ARE HIRING

4. INTRODUCTION

9. www APImobi

10. www APImobi

11. www APIfrontendbackend
mobi
events storage
events: info about
subsystems status

12. backend
web mobi api
databases
caches
load
balancersavailability
cluster
email
etc…

13. WHAT IS AN EVENT ?

14. EVENT STRUCTURE
• Provides info about subsystems
• Data
• Deep HashMap
• Timestamp
• Type + Subtype
• The rest: specific data
• Schema-less

15. EXAMPLE 1: WEB APP EVENT
• Large event
• Info about users actions
• Requests, user type
• Timings
• Warnings, errors
• Etc…

16. { timestamp => 12345,
type => 'WEB',
subtype => 'app',
action => { is_normal_user => 1,
pageview_id => '188a362744c301c2',
# ...
},
tuning => { the_request => 'GET /display/...'
bytes_body => 35,
wallclock => 111,
nr_warnings => 0,
# ...
},
# ...
}

17. EXAMPLE 2: AVAILABILITY CLUSTER EVENT
• Small event
• Cluster provides availability info
• Event: Info about request types and timings

18. { type => 'FAV',
subtype => 'fav',
timestamp => 1401262979,
dc => 1,
tuning => {
flatav => {
cluster => '205',
sum_latencies => 21,
role => 'fav',
num_queries => 7
}
}
}

19. EVENTS FLOW PROPERTIES
• Read-only
• Schema-less
• Continuous, ordered, timed
• 15 K events per sec
• 1.25 Billion events per day
• peak at 70 MB/s, min 25MB/s
• 100 GB per hour

20. SERIALIZATION
• JSON didn’t work for us (slow, big, lack features)
• Created Sereal in 2012
• « Sereal, a new, binary data serialization format that
provides high-performance, schema-less serialization »
• Added Sereal encoder & decoder in Erlang in 2014

21. USAGE

22. ASSESS THE NEEDS
• Before thinking about storage
• Think about the usage

23. USAGE
1. GRAPHS
2. DECISION MAKING
3. SHORT TERM ANALYSIS
4. A/B TESTING

24. GRAPHS
• Graph in real-time ( few seconds lag )
• Graph as many systems as possible
• General platform health check

25. GRAPHS

26. GRAPHS

27. DASHBOARDS

28. META GRAPHS

29. USAGE
1. GRAPHS
2. DECISION MAKING
3. SHORT TERM ANALYSIS
4. A/B TESTING

30. DECISION MAKING
• Strategic decision ( use facts )
• Long term or short term
• Technical / Non technical Reporting

31. USAGE
1. GRAPHS
2. DECISION MAKING
3. SHORT TERM ANALYSIS
4. A/B TESTING

32. SHORT TERM ANALYSIS
• From 10 sec ago -> 8 days ago
• Code deployment checks and rollback
• Anomaly Detector

33. USAGE
1. GRAPHS
2. DECISION MAKING
3. SHORT TERM ANALYSIS
4. A/B TESTING

34. A/B TESTING
• Our core philosophy: use facts
• It means: do A/B testing
• Concept of Experiments
• Events provide data to compare

35. EVENT AGGREGATION

36. EVENT AGGREGATION
• Group events
• Granularity we need: second

37. event
event
events storage
event
event
event
event
event
event
event
event
event
event
event

38. e e
e e
e
e
e e
e
e
e
e
e
ee
e
e
e
LOGGER
e
e

39. web api
e
e
e
e
e
e
e
e
e
e e
e e
e
e
e e
e
e
e
e
e
ee
e
e
ee
e

40. web api dbs
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e e
e e
e
e
e e
e
e
e
e
e
ee
e
e
ee
e

41. web api dbs
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e e
e e
e
e
e e
e
e
e
e
e
ee
e
e
e
1 sec
e
e

42. web api dbs
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e e
e e
e
e
e e
e
e
e
e
e
ee
e
e
e
1 sec
e
e

43. web api dbs
e
e
e
e
e
e
e
e
e
e
e
e
1 sec
events storage

44. web api dbs
e
e
e
e
e
e
e
e
e
1 sec
events storage
ee e reserialize
+ compress

45. events storage
LOGGER …LOGGER LOGGER

46. STORAGE

47. WHAT WE WANT
• Storage security
• Mass write performance
• Mass read performance
• Easy administration
• Very scalable

48. WE CHOSE RIAK
• Security: cluster, distributed, very robust
• Good and predictable read / write performance
• The easiest to setup and administrate
• Advanced features (MapReduce, triggers, 2i, CRDTs …)
• Riak Search
• Multi Datacenter Replication

50. CLUSTER
• Commodity hardware
• All nodes serve data
• Data replication
• Gossip between nodes
• No master
Ring of servers

51. hash(key)

52. KEY VALUE STORE
• Namespaces: bucket
• Values: opaque or CRDTs

53. RIAK: ADVANCED FEATURES
• MapReduce
• Secondary indexes
• Riak Search
• Multi DataCenter Replication

54. MULTI-BACKEND
• Bitcask
• Eleveldb
• Memory

55. BACKEND: BITCASK
• Log-based storage backend
• Append-only files
• Advanced expiration
• Predictable performance
• Perfect for reading sequential data

56. CLUSTER CONFIGURATION

57. DISK SPACE NEEDED
• 8 days
• 100 GB per hour
• Replication 3
• 100 * 24 * 8 * 3
• Need 60 T

58. HARDWARE
• 16 nodes
• 12 CPU cores ( Xeon 2.5Ghz)
• 192 GB RAM
• network 1 Gbit/s
• 8 TB (raid 6)
• Cluster space: 128 TB

59. RIAK CONFIGURATION
• Vnodes: 256
• Replication: n_val = 3
• Expiration: 8 days
• 4 GB files
• Compaction only when file is full
• Compact only once a day

60. DISK SPACE RECLAIMED
one day

61. DATA DESIGN

62. web api dbs
e
e
e
e
e
e
e
e
e
1 sec
events storage
1 blob per EPOCH / DC / CELL / TYPE / SUBTYPE
500 KB max chunks

63. DATA
• Bucket name: “data“
• Key: “12345:1:cell0:WEB:app:chunk0“
• Value: serialized compressed data
• About 120 keys per seconds

64. METADATA
• Bucket name: “metadata“
• Key: epoch-dc “12345-2“
• Value: list of data keys:
[ “12345:1:cell0:WEB:app:chunk0“,
“12345:1:cell0:WEB:app:chunk1“
…
“12345:4:cell0:EMK::chunk3“ ]
• As pipe separated value

65. PUSH DATA IN

66. PUSH DATA IN
• In each DC, in each cell, Loggers push to Riak
• 2 protocols: REST or ProtoBuf
• Every seconds:
• Push data values to Riak, async
• Wait for success
• Push metadata

67. JAVA
Bucket DataBucket = riakClient.fetchBucket("data").execute();
DataBucket.store("12345:1:cell0:WEB:app:chunk0", Data1).execute();
DataBucket.store("12345:1:cell0:WEB:app:chunk1", Data2).execute();
DataBucket.store("12345:1:cell0:WEB:app:chunk2", Data3).execute();
Bucket MetaDataBucket = riakClient.fetchBucket("metadata").execute();
MetaDataBucket.store("12345-1", metaData).execute();
riakClient.shutdown();

68. Perl
my $client = Riak::Client->new(…);
$client->put(data => '12345:1:cell0:WEB:app:chunk0', $data1);
$client->put(data => '12345:1:cell0:WEB:app:chunk1', $data2);
$client->put(data => '12345:1:cell0:WEB:app:chunk2', $data3);
$client->put(metadata => '12345-1', $metadata, 'text/plain' );

69. GET DATA OUT

70. GET DATA OUT
• Request metadata for epoch-DC
• Parse value
• Filter out unwanted types / subtypes
• Fetch the data keys

71. Perl
my $client = Riak::Client->new(…);
my @array = split '|', $client->get(metadata => '12345-1');
@filtered_array = grep { /WEB/ } @array;
$client->get(data => $_) foreach @array;

72. REAL TIME PROCESSING OUTSIDE OF RIAK

73. STREAMING
• Fetch 1 second every second
• Or a range ( last 10 min )
• Client generates all the epochs for the range
• Fetch all epochs from Riak

74. EXAMPLES
• Continuous fetch => Graphite ( every sec )
• Continuous fetch => Anomaly Detector ( last 2 min )
• Continuous fetch => Experiment analysis ( last day )
• Continuous fetch => Hadoop
• Manual request => test, debug, investigate
• Batch fetch => ad hoc analysis
• => Huge numbers of fetches

75. events storage
graphite
cluster
Anomaly
detector
experiment
cluster
hadoop
cluster
mysql
analysis
manual
requests
50 MB/s
50 MB/s
50
M
B/s
50
M
B/s
50 MB/s
50 MB/s

76. REALTIME
• 1 second of data
• Stored in < 1 sec
• Available after < 1 sec
• Issue : network saturation

77. REAL TIME PROCESSING INSIDE RIAK

78. THE IDEA
• Instead of
• Fetching data, crunch data, small result
• Do
• Bring code to data

79. WHAT TAKES TIME
• Takes a lot of time
• Fetching data out
• Decompressing
• Takes almost no time
• Crunching data

80. MAPREDUCE
• Send code to be executed
• Works fine for 1 job
• Takes < 1s to process 1s of data
• Doesn’t work for multiple jobs
• Has to be written in Erlang

81. HOOKS
• Every time metadata is written
• Post-Commit hook triggered
• Crunch data on the nodes

83. Riak post-commit hook
REST serviceRIAK service
key key
socket
new data sent for storage
fetch, decompress
and process all tasks
NODE HOST

84. HOOK CODE
metadata_stored_hook(RiakObject) ->
Key = riak_object:key(RiakObject),
Bucket = riak_object:bucket(RiakObject),
[ Epoch, DC ] = binary:split(Key, <<"-">>),
Data = riak_object:get_value(RiakObject),
DataKeys = binary:split(Data, <<"|">>, [ global ]),
send_to_REST(Epoch, Hostname, DataKeys),
ok.

85. send_to_REST(Epoch, Hostname, DataKeys) ->
Method = post,
URL = "http://" ++ binary_to_list(Hostname)
++ ":5000?epoch=" ++ binary_to_list(Epoch),
HTTPOptions = [ { timeout, 4000 } ],
Options = [ { body_format, string },
{ sync, false },
{ receiver, fun(ReplyInfo) -> ok end }
],
Body = iolist_to_binary(mochijson2:encode( DataKeys )),
httpc:request(Method, {URL, [], "application/json", Body},
HTTPOptions, Options),
ok.

86. REST SERVICE
• In Perl, using PSGI, Starman, preforks
• Allow to write data cruncher in Perl
• Also supports loading code on demand

87. ADVANTAGES
• CPU usage and execution time can be capped
• Data is local to processing
• Two systems are decoupled
• REST service written in any language
• Data processing done all at once
• Data is decompressed only once

88. DISADVANTAGES
• Only for incoming data (streaming), not old data
• Can’t easily use cross-second data
• What if the companion service goes down ?

89. FUTURE
• Use this companion to generate optional small values
• Use Riak Search to index and search those

90. THE BANDWIDTH PROBLEM

91. • PUT - bad case
• n_val = 3
• inside usage =
3 x outside usage

92. • PUT - good case
• n_val = 3
• inside usage =
2 x outside usage

93. • GET - bad case
• inside usage =
3 x outside usage

94. • GET - good case
• inside usage =
2 x outside usage

95. • network usage ( PUT and GET ):
• 3 x 13/16+ 2 x 3/16= 2.81
• plus gossip
• inside network > 3 x outside network

96. • Usually it’s not a problem
• But in our case:
• big values, constant PUTs, lots of GETs
• sadly, only 1 Gbit/s
• => network bandwidth issue

97. THE BANDWIDTH SOLUTIONS

98. THE BANDWIDTH SOLUTIONS
1. Optimize GET for network usage, not speed
2. Don’t choose a node at random

99. • GET - bad case
• n_val = 1
• inside usage =
1 x outside

100. • GET - good case
• n_val = 1
• inside usage =
0 x outside

101. WARNING
• Possible only because data is read-only
• Data has internal checksum
• No conflict possible
• Corruption detected

102. RESULT
• practical network usage reduced by 2 !

103. THE BANDWIDTH SOLUTIONS
1. Optimize GET for network usage, not speed
2. Don’t choose a node at random

104. • bucket = “metadata”
• key = “12345”

105. • bucket = “metadata”
• key = “12345”
Hash = hashFunction(bucket + key)
RingStatus = getRingStatus
PrimaryNodes = Fun(Hash, RingStatus)

106. hashFunction()
getRingStatus()

107. hashFunction()
getRingStatus()

108. WARNING
• Possible only if
• Nodes list is monitored
• In case of failed node, default to random
• Data is requested in an uniform way

109. RESULT
• Network usage even more reduced !
• Especially for GETs

110. CONCLUSION

111. CONCLUSION
• We used only Riak Open Source
• No training, self-taught, small team
• Riak is a great solution
• Robust, fast, scalable, easy
• Very flexible and hackable
• Helps us continue scaling

112. Q&A
@damsieboy