Acunu Analytics @ Cassandra London

Acunu Analytics
Realtime Big Data Analytics

Tom Wilkie, Acunu
16th July 2012

• Motivation / alternatives
• What is it?
• How does it work?
• Whats it good for?

2
Analytics

• What is it?

3
Analytics

time page session id duration
time page session id duration
time ...page session id duration
... time
... time
page
... page session id ......
...
... duration
... page session id duration
... time
14:58:03.234 time ...
/index.html page session id 175 ......
...
... duration
...
...
248.180.3.40 session id 175 duration
14:58:03.234 time...
14:58:03.234 time
/index.html page
...
/index.html page
248.180.3.40 session id 175 ......
...
... duration
...
14:58:03.234 /csi/csi/council/freedom.html
14:58:03.409 ... time ...
248.180.3.40
/index.html page 248.180.3.40 session id session id 175 duration
...
248.180.3.40 1234 ...
14:58:03.409 ... time /index.html page 248.180.3.40 session id duration
/csi/csi/council/freedom.html ... 248.180.3.40 1234 175 ...
...
/index.html page 248.180.3.40 session id duration
14:58:03.234
/docs/access/chapter8.txt ...... page 248.180.3.40 ...session id ......
14:58:03.409 ... time 248.180.3.40 1234 175 duration
/csi/csi/council/freedom.html 99.1.10.178 52
/docs/access/chapter8.txt ... page 248.180.3.40 ...session id duration
14:58:03.409 ... time
14:58:03.877 14:58:03.234 /index.html 248.180.3.40 1234 175
14:58:03.877 14:58:03.234 /csi/csi/council/freedom.html
14:58:03.409 ... time /index.html 99.1.10.178 52
248.180.3.40 1234 175 ...
... 52 1234 175 duration
14:58:03.409 ... /csi/csi/council/freedom.html 99.1.10.178
/docs/access/chapter8.txt ... page 248.180.3.40 session id
14:58:03.87714:58:03.234 time 248.180.3.40 ...session id ......
248.180.3.40 1234 175 duration
14:58:03.877 /index.html 248.180.3.40
14:58:03.877 /docs/access/chapter8.txt
14:58:03.409 ... time/docs/access/chapter8.txt ...99.1.10.178
/csi/csi/council/freedom.html 99.1.10.178
/index.html page 52 ...52
248.180.3.40 session id duration
14:58:03.234
14:58:03.877 14:58:03.234 time /docs/access/chapter8.txt ...99.1.10.178
/index.html page 52 ... 1234 175 ...
52
52 ... 1234 175 duration
248.180.3.40
14:58:03.409 ...
/docs/access/chapter8.txt
/docs/access/chapter8.txt ...99.1.10.178 99.1.10.178 52
14:58:03.87714:58:03.409 ...... /csi/csi/council/freedom.html 99.1.10.17852 52session id 175 ......
/docs/access/chapter8.txt /index.html page 248.180.3.40
14:58:03.877 14:58:03.234 time /csi/csi/council/freedom.html
14:58:03.409 /docs/access/chapter8.txt 99.1.10.178 248.180.3.40 session id duration
99.1.10.178 248.180.3.40... 1234
14:58:03.877
/docs/access/chapter8.txt 99.1.10.178 52
/docs/access/chapter8.txt ......99.1.10.178
14:58:03.877 14:58:03.234 time
14:58:03.877 14:58:03.409 ... /csi/csi/council/freedom.html 99.1.10.17852 52 ... 1234 175 duration
/docs/access/chapter8.txt /index.html page 248.180.3.40
14:58:03.877
14:58:03.877 /docs/access/chapter8.txt 52
14:58:03.877 14:58:03.409 ... /csi/csi/council/freedom.html 99.1.10.17852 52 ... 1234 175 ......
14:58:03.877 14:58:03.234 248.180.3.40
/docs/access/chapter8.txt /index.html 99.1.10.178 248.180.3.40 52
99.1.10.178
14:58:03.877 /docs/access/chapter8.txt248.180.3.40
14:58:03.877 14:58:03.234 /docs/access/chapter8.txt ... 248.180.3.40
14:58:03.409 14:58:03.877 /docs/access/chapter8.txt
14:58:03.409 ... /docs/access/chapter8.txt ... 1234
14:58:03.877 14:58:03.234 /csi/csi/council/freedom.html 99.1.10.17852 52 ... 1234 175 ...
99.1.10.178 248.180.3.40
/index.html 99.1.10.178 248.180.3.40
/csi/csi/council/freedom.html
/docs/access/chapter8.txt /index.html
14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40
14:58:03.877 14:58:03.234 /csi/csi/council/freedom.html 99.1.10.17852 52 52 1234 175
99.1.10.178 1234 248.180.3.40
248.180.3.40
14:58:03.877 /csi/csi/council/freedom.html 248.180.3.40
/docs/access/chapter8.txt /index.html 99.1.10.178
14:58:03.409 14:58:03.877 /docs/access/chapter8.txt 99.1.10.178 1234 52
/docs/access/chapter8.txt/csi/csi/council/freedom.html 99.1.10.17852 52 52 1234 175
14:58:03.87714:58:03.877 14:58:03.234 /csi/csi/council/freedom.html 99.1.10.178 248.180.3.40 1234 175
14:58:03.409 /docs/access/chapter8.txt 99.1.10.17852248.180.3.40
14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40 1234 52
/csi/csi/council/freedom.html 99.1.10.178 248.180.3.40
14:58:03.877 /docs/access/chapter8.txt 99.1.10.178 99.1.10.178
14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 99.1.10.17852248.180.3.40
14:58:03.87714:58:03.877 14:58:03.234 /csi/csi/council/freedom.html 99.1.10.17852 52 52 1234
1234
14:58:03.877 /csi/csi/council/freedom.html 99.1.10.17852248.180.3.40
/docs/access/chapter8.txt/csi/csi/council/freedom.html 99.1.10.17852 52 52 1234
1234
14:58:03.409 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.178 99.1.10.178 248.180.3.40
14:58:03.409 /docs/access/chapter8.txt 99.1.10.17852248.180.3.40
/csi/csi/council/freedom.html/docs/access/chapter8.txt 1234 99.1.10.17852 52 52 1234
248.180.3.40
/docs/access/chapter8.txt/csi/csi/council/freedom.html
14:58:03.409 14:58:03.877 /docs/access/chapter8.txt 1234
99.1.10.178
14:58:03.409 14:58:03.87714:58:03.409
14:58:03.877 /csi/csi/council/freedom.html/docs/access/chapter8.txt 1234 99.1.10.17852 52 52
/docs/access/chapter8.txt 248.180.3.40
99.1.10.178
14:58:03.409 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.178 99.1.10.178 1234 52 52 52
14:58:03.87714:58:03.877
/csi/csi/council/freedom.html/docs/access/chapter8.txt
99.1.10.178 248.180.3.40 99.1.10.178
/docs/access/chapter8.txt /docs/access/chapter8.txt 99.1.10.178 99.1.10.17852 52 52
14:58:03.877 /docs/access/chapter8.txt 99.1.10.17852 99.1.10.178
14:58:03.877 14:58:03.409 14:58:03.877
14:58:03.409 14:58:03.877 /csi/csi/council/freedom.html 248.180.3.40 99.1.10.178
14:58:03.877 /csi/csi/council/freedom.html/docs/access/chapter8.txt 1234 52 1234
14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40 99.1.10.178 52 1234 52 52
14:58:03.87714:58:03.877
14:58:03.409 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.17852 1234
/docs/access/chapter8.txt /docs/access/chapter8.txt
14:58:03.87714:58:03.877
14:58:03.87714:58:03.877
14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40 99.1.10.178 52 1234 52
14:58:03.87714:58:03.877
14:58:03.877
/docs/access/chapter8.txt /docs/access/chapter8.txt 248.180.3.40
14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40 99.1.10.178 52 1234
/csi/csi/council/freedom.html 99.1.10.17852 1234
248.180.3.40
/csi/csi/council/freedom.html 99.1.10.17852 1234 52 1234
14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40 248.180.3.40
14:58:03.877 /csi/csi/council/freedom.html 99.1.10.17852 1234 52 1234
248.180.3.40
14:58:03.877 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.178 99.1.10.17852 1234 52
14:58:03.877 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.178 99.1.10.17852 1234 52
14:58:03.409 99.1.10.17899.1.10.17852 1234 52
248.180.3.40 52 1234
248.180.3.40 52 1234
14:58:03.877
248.180.3.40 52 1234
14:58:03.877 /docs/access/chapter8.txt 99.1.10.178 52

4
Analytics

Combining “big” and “real-time” is hard

Live & historical Drill downs
Trends...
aggregates... and roll ups

5
Analytics

Solution Con

Scalability
$$$

Not realtime
Inefﬁcient Recomputation

Spartan query semantics =>
complex, DIY solutions

6
Analytics

• What is it?

7
Analytics

Analytics

counter
updates
Click stream events
Acunu
Sensor data
Analytics
etc

• Simple, real-time, incremental analytics
• Push processing into ingest phase

{
time : TIME(HOUR; MIN; SEC),
page : PATH(/),
category : STRING,
loadTime : LONG
}

{
select : ["COUNT", "AVG(loadTime)"],
where : “time, ?path”,
group : “time, ?category”
}

9
Analytics

• What is it?

10
Analytics

Introduction

11
Analytics

count
grouped by ...
day
count
distinct
(session)
count ... geography

avg(duration)
... browser

12
Analytics

time : TIME(HOUR; MIN; SEC),
cust_id : LONG,
Data session_id : LONG,
Deﬁnition geography : STRING,
browser : STRING,
load_time : LONG

{ select: “COUNT”
patterns: [
{ where : “?time”, group : “?time” },

Query { where : “”, group : “geography” },
{ where : “”, group : “browser” }
Patterns ]
}, {
select: [“COUNT_DISTINCT(session_id)”,
“AVG(load_time)”],
where: “time”, group: “”
}

13
Analytics

21:00 all→1345 :00→45 :01→62 :02→87 ...

22:00 all→3221 :00→22 :00→19 :02→104 ...
{
cust_id: user01, ... ...

session_id: 102, UK all→228 user01→1 user14→12 user99→7 ...
geography: UK,
US all→354 user01→4 user04→8 user56→17 ...
browser: IE,
time: 22:02, ...

} UK, 22:00 all→1904 ...

∅ all→87314 UK→238 US→354 ...

14
Analytics

21:00 all→1345 :00→45 :01→62 :02→87 ...

22:00 all→3222 :00→22 :00→19 :02→105 ...
{
cust_id: user01, ... ...

session_id: 102, UK all→229 user01→2 user14→12 user99→7 ...
geography: UK,
US all→354 user01→4 user04→8 user56→17 ...
browser: IE,
time: 22:02, ...

} UK, 22:00 all→1905 ...

∅ all→87315 UK→239 US→354 ...

15
Analytics

where time 21:00-22:00
count(*)
21:00 all→1345 :00→45 :01→62 :02→87 ...

where time 22:00-23:00, 22:00 all→3222 :00→22 :01→19 :02→105 ...

group by minute ... ...

UK all→229 user01→2 user14→12 user99→7 ...

where geography=UK US all→354 user01→4 user04→8 user56→17 ...

group all by user, ...

UK, 22:00 all→1905 ...

count all ∅ all→87315 UK→239 US→354 ...

group all by geo
16
Analytics

• SUM, COUNT, MIN, MAX, STDDEV, AVG,
TOP k, COUNT DISTINCT
• Also: approx top k, approx count distinct
• Also: idempotent update
• RESTful JSON interface, CLI

17
Analytics

• What is it?

18
Analytics

Manufacturing Social Media Ad Analytics

Systems Financial
Oil + Gas
Monitoring Services

Analytics

“We keep discovering use cases
we hadn’t thought of ”

“We found out a competitor
was scraping our data”

“Up and running in about 4 hours”

Analytics

Thanks!
Questions?

analytics@acunu.com

21
Analytics

Acunu Analytics @ Cassandra London

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (17)

Mehr von Acunu

Mehr von Acunu (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Acunu Analytics @ Cassandra London