Presentation about Big Data Adoption Status by Nuno Barreto - Partner & Big Data Lead @Xpand IT

2. Big Data Adoption Status
Nuno Barreto
Partner | Big Data Lead

3. AGENDA
• STATUS CHECK
• TECHNOLOGY HIGHLIGHTS
• COOL STUFF WE’RE DOING
• LOOKING AHEAD

4. STATUS CHECK

5. HADOOP – THE ULTIMATE DATA TOOLKIT

6. HADOOP – THE ECO-SYSTEM

7. from Batch to Near-Real-Time
from Analytics to Operational

8. ACTIVE INDUSTRIES
• RETAIL
• UTILITIES
• TELCO
• MOBILITY
• FINANCIAL SERVICES
• E-BUSINESS

9. CLUSTER SIZES & TOPOLOGIES
• 5 TO TENS OF NODES
• COUPLE HUNDRED GiB TO DUZEN
TiB OF RAM
• COUPLE TiB TO HUNDRED TiB OF
RAW SPACE
• ON-PREM AND CLOUD

11. TECHNOLOGY HIGHLIGHTS

13. HBASE AND HDFS/PARQUET
• HDFS/PARQUET IS
GREAT FOR
LARGE SCANS,
BUT…
• HBASE IS GREAT
FOR INDEXED
READS/WRITES,
BUT…

14. COMPLEX ARCHITECTURES
source: http://blog.cloudera.com/blog/2015/09/kudu-new-apache-hadoop-storage-for-fast-analytics-on-fast-data/

15. THE “WORST” OF BOTH WORLDS
• KUDU IS SLIGHTLY
WORSE THAN
HBASE FOR
INDEXED OPS
• KUDU SHOULD BE
NO MORE THAN 2x
WORSE THAN
PARQUET FOR
LARGE SCANS

16. KUDU BENCHMARKS – INGEST RATE
HIGHER IS BETTER
source: http://blog.cloudera.com/blog/2017/02/performance-comparing-of-different-file-formats-and-storage-engines-in-hadoop-file-system/

17. KUDU BENCHMARKS – RANDOM LOOKUP
LOWER IS BETTER
source: http://blog.cloudera.com/blog/2017/02/performance-comparing-of-different-file-formats-and-storage-engines-in-hadoop-file-system/

18. KUDU BENCHMARKS – SCAN RATE
HIGHER IS BETTER
source: http://blog.cloudera.com/blog/2017/02/performance-comparing-of-different-file-formats-and-storage-engines-in-hadoop-file-system/

19. KUDU BENCHMARKS – SUMMARY
source: http://blog.cloudera.com/blog/2017/02/performance-comparing-of-different-file-formats-and-storage-engines-in-hadoop-file-system/

20. COOL STUFF
WE’RE DOING

21. NEAR–REAL–TIME DATA LAKE

22. BY

23. ZWOOX - INGESTION FRAMEWORK
• LOW LATENCY, HIGH THROUGHPUT, HIGHLY
AVAILABLE
• NEAR-REAL TIME for KUDU & BATCH for HIVE/IMPALA
• BATCH & STREAMING REPLICATIONS
• AUTOMATIC CONSOLIDATION INTO HDFS BASED
TABLES
• MULTIPLE TABLEs WITH SPECIFIC PARTITIONING
SCHEME
• IN-LINE PROCESSING
• AUTOMATIC AUDIT DATA

24. MESSAGE BUS
• JMS SEMANTICS IS LIMITED
• JMS SCALING IS HARD
• JMS PERFORMANCE IS POOR COMPARED TO KAFKA

25. LOOKING AHEAD
(two trends)

26. IOT – VALUE IS IN THE CORE
DEVICE CENTRIC
VS
GATEWAY CENTRIC
VS
CENTRALIZED
PLATFORM
CENTRIC

27. DATA SCIENCE

28. CLOUDERA WORKBENCH

29. THANK YOU