Date: 16th November 2017 Location: Fast Data Theatre Time: 15:10 - 15:40 Speaker: Keith Bolam Organisation: Actian

1. One Billion rows
Fresh and Fast Data
November 16, 2017
KEITH BOLAM
ENGINERING SOLUTIONS MANAGER

2. FIVE WS AND H
I keep six honest serving-men
(They taught me all I knew);
Their names are What and Why and When
And How and Where and Who.
(Kipling 1902)

3. Your analytics strategy
cost-effective
performance that meets the
speed of your business
on-premise or cloud
deployment
accessible using tools and
skills that are available
Bring new capabilities -
predictive analytics and ML

4. Where should I be looking?
Am I Looking hard
in the wrong
place?
Consumes
resources
Valid projects may
be abandoned
Brings about
missed
opportunities
© 2017 Actian Corporation4

5. What am I looking at?
Am I looking the
right way ?
Look where the
fish are!
© 2017 Actian Corporation5

6. Why do I need help?
A helping hand
can direct the way
to look and where.
Changing the
paradigm is not
always easy.
Help is often
required from
outside the
organization.
© 2017 Actian Corporation6

7. When does the Eureka moment appear?
Tall trees.
Stable,
unchanging,
dependable…
Or are they?
What-If we want
to prove this.
© 2017 Actian Corporation7

8. When does the Eureka moment appear?
Tall trees.
Stable,
unchanging,
dependable…
Or are they?
What-If we want
to prove this.
Is this the right
sample size?
© 2017 Actian Corporation8

9. When do I need to Focus?
Tall trees.
Stable,
unchanging,
dependable…
Or are they?
What-If we want
to prove this.
Is this the right
sample size?
Need to Focus.
© 2017 Actian Corporation9

10. A ‘database’ started in 1936 of California Redwoods
Measuring trees !
This is a page from
1959
© 2017 Actian Corporation10

11. A ‘database’ started in 1936 of California Redwoods
Measuring trees !
There are rule
Rules exist to
make comparison
possible
Do’s and Do not’s
© 2017 Actian Corporation11

12. How do I know that this is right?
Tall trees.
Stable,
unchanging,
dependable…
Or are they?
What-If we want
to prove this.
Need to Focus.
And pay attention
to detail.
© 2017 Actian Corporation12

13. What am I looking at?
Chimneys are tall
Generally like tall
trees
– have a fluffy
‘leaf’ like
proliferation at
the top
© 2017 Actian Corporation13

14. How do I Focus?
Chimneys are tall
Generally like tall
trees
– have a fluffy
‘leaf’ like
proliferation at
the top
– But sometimes
they do not
© 2017 Actian Corporation14

15. How do I Focus?
Chimneys are tall
Generally like tall
trees
– have a fluffy ‘leaf’
like proliferation
at the top
– But sometimes
they do not
Chimneys may not
really be chimneys
– Pay attention to
detail
– Tall, straight,
fluffy cloud
© 2017 Actian Corporation15

16. What am I looking at?
Wind Turbines are
tall
Generally like tall
trees
– have a fluffy ‘leaf’
like proliferation
at the top
© 2017 Actian Corporation16

17. What am I looking at?
Wind Turbines are
tall
Generally like tall
trees
– have a fluffy ‘leaf’
like proliferation
at the top
– But not
necessarily easy
to get to
© 2017 Actian Corporation17

18. Time to Focus?
Wind Turbines are tall
Generally like tall
trees
– have a fluffy ‘leaf’
like proliferation at
the top
– Can be disguised –
or made to look
different
– There could be a lot
of other
measurements we
could
What measurements
could we consider
© 2017 Actian Corporation18

19. However beware – all Windmills are not equal.
Wind Turbines are
tall
Generally like tall
trees
– have a fluffy ‘leaf’
like proliferation
at the top
– Can be disguised
– or made to look
different
© 2017 Actian Corporation19

20. However beware – all Wind Farm’s are not equal!
Wind Turbines are
tall
Generally like tall
trees
– have a fluffy ‘leaf’
like proliferation
at the top
– Can be disguised
– or made to look
different
Wind Turbine
Farms
– Hmm…
© 2017 Actian Corporation20

21. However beware – all Wind Farm’s are not equal!
Wind Turbines are
tall
Generally like tall
trees
– have a fluffy ‘leaf’
like proliferation
at the top
– Can be disguised
– or made to look
different
Wind Turbine
Farms
– Hmm…
© 2017 Actian Corporation21

22. Where should we be focusing?
Are we looking at a
constant flow?
Is the bridge
crossing the gorge?
© 2017 Actian Corporation22

23. Where should we be focusing?
Are we looking at a
constant flow?
Is the bridge
crossing the gorge?
Or…
Is the bridge a
vantage point to
observe the flow?
© 2017 Actian Corporation23

24. Where should we be focusing?
London’s green
spaces
700k trees in public
spaces that have
been mapped
More than 2m
others are in private
gardens or parks
© 2017 Actian Corporation24

25. Where should we be focusing.
London’s green
spaces
700k trees in public
spaces mapped
Each borough has
its own database.
© 2017 Actian Corporation25

26. Where should we be focusing.
London’s green
spaces
700k trees in public
spaces mapped
Each borough has
its own database.
Pear Trees!
© 2017 Actian Corporation26

27. One Billion rows
Actian#1 Vector
(#1 Pronounced AK-TI-AN)

28. Actian Delivers The Broadest Range of Analytic Engines
Vectorized
– Uses special instruction set in industry-standard CPUs to accelerate
performance, without tuning or special-purpose hardware
Columnar
– Formats data for faster scanning, fewer transfers to/from disk, reduced I/O, and
better compression
In-memory
– Takes better advantage of cache architecture in modern CPUs to further
accelerate performance. Available SINCE 2000
SQL
– Fully supports industry standard ANSI SQL 2003 so your staff can use the tools
and interfaces they know already
Database
– Available on Linux and Windows, one server or the largest cluster, integrated
with open source software like Hadoop and Spark
© 2017 Actian Corporation28

29. Database Node
Actian Vector Analytics Family Overview
© 2017 Actian Corporation29
Actian Vector
World’s Fastest
Analytic Database
Vectorized Columnar Analytic Database
Column storage
for Analytics
X100

30. X100X100
X100X100
Database Node
Actian Vector Analytics Family Overview
© 2017 Actian Corporation30
Actian Vector in Hadoop
Scaling Vector Performance
To Multi-node Clusters
Actian Vector
World’s Fastest
Analytic Database
Vectorized Columnar Analytic Database
Column storage
for Analytics
Clustered Analytic Database
X100

31. X100X100
X100X100
Database NodeDatabase Node
Actian Vector Analytics Family Overview
© 2017 Actian Corporation31
Actian X
Linking Operational and
Analytic Databases
Hybrid OLTP & Analytic Database
Row storage
for OLTP
Column storage
for Analytics
Actian Vector in Hadoop
Scaling Vector Performance
To Multi-node Clusters
Actian Vector
World’s Fastest
Analytic Database
Vectorized Columnar Analytic Database
Column storage
for Analytics
Clustered Analytic Database
X100OLTPX100

32. Actian Vector X100 Analytics Engine - The Secret
Ingredient
 Top performance in industry-standard
TPC-H benchmark
 Key innovations:
– Exploits Intel’s vector instruction set to
process more data elements per
instruction
– Columnar data format to reduce data
transfers from disk and improve
compression optimization
– Optimizes use of large CPU caches to
beat simple in-memory databases
– Tracks write activity to allow updates
without impacting performance,
maintains full ACID compliance via
WAL and PDTs
– Tracks block metadata to avoid
unnecessary data transfers from disk

33. Actian Vector X100 Analytics Engine - The Secret
Ingredient

34. Vector Processing Provides Performance Advantage
 Intel Advanced Vector Extensions enable Single Instruction Multiple Data
(SIMD) processing to improve throughput
 SIMD processes multiple pieces of data in a single step, speeding up
throughput for many tasks, including data analysis
 X100: first database engine to be designed from the ground up to exploit SIMD
technology, years ahead
 Vector operates by treating data in the registers in parallel “SIMD” mode; the
scalar version only operates on one entry in each register. SIMD maximizes use
of registers and minimizes data movement, providing better overall
throughput.
© 2017 Actian Corporation34

35. Actian Vector (VectorH) to Scale Out
© 2017 Actian Corporation35
 Vectorized, columnar, in-memory
analytic database that runs within a
Hadoop cluster
 Supported on Hortonworks, Cloudera,
and Apache Hadoop distributions
 Supported on Linux and Windows
 Linear scalability to TBs and PBs of
data with more nodes
 Available in Trial and Enterprise
editions
– Trial edition provides a 30-day license with all
the features of Enterprise to be used for testing
and proofs of concept
– Enterprise edition has annual or perpetual
license options with full Enterprise support, no
storage limit
Deploy on your choice of cloud
including Amazon or Microsoft
Azure
Integrated with Spark for data
transformation and to accelerate
Spark workloads like machine
learning and graph analytics

36. create table actian.yellow_tripdata_staging2_vw as
select year(tpep_dropoff_datetime) as "year",
month(tpep_dropoff_datetime) as "month",
day(tpep_dropoff_datetime) as "day",
hour(tpep_dropoff_datetime) as "hour",
TO_CHAR(tpep_dropoff_datetime,'YYYYMM') as "yearmonth",
tpep_dropoff_datetime, tpep_pickup_datetime,
passenger_count, trip_distance ,
pickup_location_id , dropoff_location_id , fare_amount ,
time(timestamp(tpep_pickup_datetime),0) as "Starttime",
time(timestamp(tpep_dropoff_datetime),0) as "Endtime",
vendor_id
FROM actian.yellow_tripdata_staging2
WHERE pickup_location_id IS NOT NULL
24 real cores / 48 threads
KiB Mem : 26385971
22 500GB 15k spinning disks RAID 5
Actian Vector X100 Analytics Engine - The Secret
Ingredient

37. create table actian.yellow_tripdata_staging2_vw as
select year(tpep_dropoff_datetime) as "year",
month(tpep_dropoff_datetime) as "month",
day(tpep_dropoff_datetime) as "day",
hour(tpep_dropoff_datetime) as "hour",
TO_CHAR(tpep_dropoff_datetime,'YYYYMM') as "yearmonth",
tpep_dropoff_datetime, tpep_pickup_datetime,
passenger_count, trip_distance ,
pickup_location_id , dropoff_location_id , fare_amount ,
time(timestamp(tpep_pickup_datetime),0) as "Starttime",
time(timestamp(tpep_dropoff_datetime),0) as "Endtime",
vendor_id
FROM actian.yellow_tripdata_staging2
WHERE pickup_location_id IS NOT NULL
Executing . . .
(0.000504 secs)
(120737401 rows in 300.498619 secs)
continue
24 real cores / 48 threads
KiB Mem : 26385971
22 500GB 15k spinning disks RAID 5
Actian Vector X100 Analytics Engine - The Secret
Ingredient

38. select count(*)
from yellow_tripdata_staging2_vwg
Executing . . .
+----------------------+
|col1 |
+----------------------+
| 1210737401|
+----------------------+
(1 row in 0.055525 secs)
X100 Analytics Engine - The Secret Ingredient

39. select count(*)
from yellow_tripdata_staging2_vwg
Executing . . .
+----------------------+
|col1 |
+----------------------+
| 1210737401|
+----------------------+
(1 row in 0.055525 secs)
select (select count(*)/1024/1024 from
yellow_tripdata_staging2) as row_count_m,
(select count(*) from yellow_tripdata_staging2 ) as
row_count
Executing . . .
+--------------+------------------+
|row_count_m |row_count |
+--------------+------------------+
| 1271| 1333572558|
+--------------+------------------+
(1 row in 0.064329 secs)
X100 Analytics Engine - The Secret Ingredient

40. Who needs this kind of performance?
© 2017 Actian Corporation40
Here we are looking at a raw taxi
data table and pulling distinct
values out. 120m tuple table
Only two years exist
All 12 months and 31 days are
present
The data is for one hour of the day
There are 10 users ‘types’ business,
personal etc

41. What does the performance give us?
© 2017 Actian Corporation41
Here we are looking at a raw taxi
data table and pulling distinct
values out. 120m tuple table
Only two years exist
All 12 months and 31 days are
present
The data is for one hour of the day
There are 10 users ‘types’ business,
personal etc

42. © 2017 Actian Corporation42

43. One Billion Rows
© 2017 Actian Corporation43
Looking at the data
1.1 Bn Tuples
5 years of historic data
 yellow taxi data (New York)
 URL accessible data
https://s3.amazonaws.com/nyc-
tlc/trip+data
City bike data (Ney York)
https://github.com/toddwschneid
er/nyc-citibike-data
cccc

44. One Billion Rows
© 2017 Actian Corporation44
cccc

45. Actian Vector Analytics make the impossible possible
Confidential © 2017 Actian Corporation45
Record-Setting
Performance
World’s Fastest Analytic Database
Actian Vector
Analytic Database

46. Actian Vector Analytics make the impossible possible
Confidential © 2017 Actian Corporation46
Record-Setting
Performance
Scales Out on
Hadoop Clusters
World’s Fastest Analytic Database
Actian Vector
Analytic Database
Actian Vector in
HadoopAnalytic
Database

47. Actian Vector Analytics make the impossible possible
Confidential © 2017 Actian Corporation47
Record-Setting
Performance
Scales Out on
Hadoop Clusters
Fast Transactions
and Analytics in One
World’s Fastest Analytic Database
Actian X Hybrid
Database
Actian Vector
Analytic Database
Actian Vector in
HadoopAnalytic
Database

48. © 2017 Actian Corporation48
For Details and follow-up
please come along to Booth 426
or contact me
keith.bolam@actian.com
THANK YOU!

49. © 2017 Actian Corporation49

50. Acknowledgements