1. USING NODEJS TO COUNT 30
BILLION REQUESTS PER DAY
#NINGJS2016
GUILLAUME VERBAL @DOWEIG
2. USING NODEJS TO COUNT 30 BILLION REQUESTS PER DAY
WHO WE ARE
▸ We are mainly an advertising company
▸ Doing many more !
▸ networks, data collection/analysis, web tech, app generation…
▸ 250 employees and counting, (100 tech people)
▸ Based in the center of the world (Beijing)
▸ I am:
▸ french, living in Beijing
▸ a software engineer (5 years) wannabe data scientist
▸ really into the blockchain
8. WE JUST PUSHED SOMETHING ONLINE
THE WORLD IS FULL OF QUESTIONS
▸ Is it crashing?
▸ Is it fast?
▸ Is our feature awesome yet?
▸ Are we famous yet?
▸ Are we rich yet?
9. WE JUST PUSHED SOMETHING ONLINE
OUR APP THROUGH THE NAKED EYE
34. WHAT IS AGGREGATION
id: 2,
browser: “IE”,
uri: “/login”,
timestamp: Date
id: 4,
browser: “IE”,
uri: “/”,
timestamp: Date
id: 1,
browser: “Safari”,
uri: “/”,
timestamp: Date
id: 3,
browser: “Chrome”,
uri: “/home”,
timestamp: Date
LEARNING THROUGH EXAMPLE CLIENT
BROWSER
SERVER
RESPONSE
REQUEST
35. WHAT IS AGGREGATION
id: 2,
browser: “IE”,
uri: “/login”,
timestamp: Date
id: 4,
browser: “IE”,
uri: “/”,
timestamp: Date
id: 1,
browser: “Safari”,
uri: “/”,
timestamp: Date
id: 3,
browser: “Chrome”,
uri: “/home”,
timestamp: Date
LEARNING THROUGH EXAMPLE CLIENT
BROWSER
SERVER
RESPONSE
REQUEST
36. WHAT IS AGGREGATION
AGGREGATION BUCKETS
id: 2,
browser: “IE”,
uri: “/login”,
timestamp: Date
id: 4,
browser: “IE”,
uri: “/”,
timestamp: Date
id: 1,
browser: “Safari”,
uri: “/”,
timestamp: Date
id: 3,
browser: “Chrome”,
uri: “/home”,
timestamp: Date
37. WHAT IS AGGREGATION
AGGREGATION BUCKETS
id: 2,
browser: “IE”,
uri: “/login”,
timestamp: Date
id: 4,
browser: “IE”,
uri: “/”,
timestamp: Date
id: 1,
browser: “Safari”,
uri: “/”,
timestamp: Date
id: 3,
browser: “Chrome”,
uri: “/home”,
timestamp: Date
SAFARI IE CHROME
38. WHAT IS AGGREGATION
AGGREGATION BUCKETS
id: 2,
browser: “IE”,
uri: “/login”,
timestamp: Date
id: 4,
browser: “IE”,
uri: “/”,
timestamp: Date
id: 1,
browser: “Safari”,
uri: “/”,
timestamp: Date
id: 3,
browser: “Chrome”,
uri: “/home”,
timestamp: Date
SAFARI IE CHROME
39. WHAT IS AGGREGATION
AGGREGATION BUCKETS
id: 2,
browser: “IE”,
uri: “/login”,
timestamp: Date
id: 4,
browser: “IE”,
uri: “/”,
timestamp: Date
id: 1,
browser: “Safari”,
uri: “/”,
timestamp: Date
id: 3,
browser: “Chrome”,
uri: “/home”,
timestamp: Date
SAFARI IE CHROME
1
40. WHAT IS AGGREGATION
AGGREGATION BUCKETS
id: 2,
browser: “IE”,
uri: “/login”,
timestamp: Date
id: 4,
browser: “IE”,
uri: “/”,
timestamp: Date
id: 1,
browser: “Safari”,
uri: “/”,
timestamp: Date
id: 3,
browser: “Chrome”,
uri: “/home”,
timestamp: Date
SAFARI IE CHROME
1 2
41. WHAT IS AGGREGATION
AGGREGATION BUCKETS
id: 2,
browser: “IE”,
uri: “/login”,
timestamp: Date
id: 4,
browser: “IE”,
uri: “/”,
timestamp: Date
id: 1,
browser: “Safari”,
uri: “/”,
timestamp: Date
id: 3,
browser: “Chrome”,
uri: “/home”,
timestamp: Date
SAFARI IE CHROME
1 2 1
52. ELK
BASIC CONCEPTS
▸ Our app will send every log to elastic search (using logstash)
▸ We will use kibana to send aggregation queries to elastic search.
▸ Kibana will make the aggregation response very visual
